Can anyone help with how to attack this question? thanks!
Make the substitutions $\displaystyle u=x^2+y^2$ and $\displaystyle v=x^2-y^2$. (Also keep in mind that with these substitutions, $\displaystyle 4\leq u\leq9$ and $\displaystyle 1\leq v\leq2$)
It follows then that $\displaystyle u-x^2=y^2$ and $\displaystyle x^2-v=y^2$
Therefore, $\displaystyle u-x^2=x^2-v\implies \tfrac{1}{2}(u+v)=x^2\implies x=\sqrt{\tfrac{1}{2}(u+v)}$
Similarly, $\displaystyle u-y^2=x^2$ and $\displaystyle v+y^2=x^2$
Therefore, $\displaystyle u-y^2=v+y^2\implies \tfrac{1}{2}(u-v)=y^2\implies y=\sqrt{\tfrac{1}{2}(u-v)}$
Now, we need to find the Jacobian:
$\displaystyle J(u,v)=\begin{vmatrix}\displaystyle\frac{\partial x}{\partial u} &\displaystyle\frac{\partial x}{\partial v}\\\displaystyle\frac{\partial y}{\partial u} & \displaystyle\frac{\partial y}{\partial v}\end{vmatrix}=\begin{vmatrix}\displaystyle\frac{ 1}{2\sqrt{2(u+v)}} & \displaystyle\frac{1}{2\sqrt{2(u+v)}}\\\displaysty le\frac{1}{2\sqrt{2(u-v)}}&\displaystyle-\frac{1}{2\sqrt{2(u-v)}}\end{vmatrix}$ $\displaystyle =-\frac{1}{8\sqrt{u^2-v^2}}-\frac{1}{8\sqrt{u^2-v^2}}=-\frac{1}{4\sqrt{u^2-v^2}}$
Therefore, $\displaystyle \iint\limits_{S}xy\,dA\implies\int_1^2\int_4^9\sqr t{\tfrac{1}{2}(u+v)}\sqrt{\tfrac{1}{2}(u-v)}\cdot\left|-\frac{1}{4\sqrt{u^2-v^2}}\right|\,du\,dv=\tfrac{1}{8}\int_1^2\int_4^9\ ,du\,dv$
(Since we're looking for the area in the first quadrant, we might have to divide our answer by 4).
I'm sure you can take it from here...
Those circles are represented by $\displaystyle 4\le x^2+y^2\le9$ so $\displaystyle 4\le u\le9.$
Besides, use
$\displaystyle \frac{\partial (x,y)}{\partial (u,v)}=\frac{1}{\dfrac{\partial (u,v)}{\partial (x,y)}},$
so there's no need to isolate variables.