I am trying to apply the Rayleigh quotient iteration to the matrix A=
where a is different from b (a and b are reals).
Find the subsets S of R^2(Reals^2) having the property that the iteration applied to this matrix with initial guesses in S do not converge. Is S a set of measure 0?
I am not sure how to proceed here. Any help is much appreciated.