$\displaystyle \int_{0}^{5}\int_{0}^{\sqrt{{25-x^{2}}}}ln(1+x^{2}+y^{2})dxdy$
how to solve it?
First, it should be dy dx not dx dy in your integral.
No need for the jocabian here.
For polar coordinates, dx dy and dy dx = $\displaystyle r \, dr \, d\theta$
Your region is $\displaystyle D=(x,y) : 0 \leq x \leq 5$ and $\displaystyle 0 \leq y \leq \sqrt{25-x^2}$
Your integral in polar coordinates is : $\displaystyle \int \int_R r \, ln(r^2+1) \, dr \, d\theta$
where $\displaystyle R=(r,\theta) : \alpha \leq \theta \leq \beta$ and $\displaystyle a \leq r \leq b$
To find $\displaystyle \alpha , \beta , a$ and $\displaystyle b$.., Draw D and convert it to polar coordinates system.
How in the world did you get that? As you say, r= x^2+ y^2 so sqrt(25- x^2- y^2) is sqrt(25- r^2). There is no "cos^2(theta)". (That would be just 25- x^2.)
And you should not need to worry about the Jacobian- when you first learned to integrate in polar coordinates you should have learned that dxdy becomes r dr dtheta.
(That is, of course, because the Jacobian is "r":
$\displaystyle \left|\begin{array}{cc}\frac{\partial x}{\partial r} & \frac{\partial y}{\partial r} \\ \frac{\partial x}{\partial \theta} & \frac{\partial y}{\partial \theta}\end{array}\right|$$\displaystyle = \left|\begin{array}{cc} cos(\theta) & sin(\theta) \\ -r sin(\theta) & r cos(\theta)\end{array}\right|= r cos^2(\theta)+ r sin^2(\theta)= r$)