how to solve it?
First, it should be dy dx not dx dy in your integral.
No need for the jocabian here.
For polar coordinates, dx dy and dy dx =
Your region is and
Your integral in polar coordinates is :
where and
To find and .., Draw D and convert it to polar coordinates system.
How in the world did you get that? As you say, r= x^2+ y^2 so sqrt(25- x^2- y^2) is sqrt(25- r^2). There is no "cos^2(theta)". (That would be just 25- x^2.)
And you should not need to worry about the Jacobian- when you first learned to integrate in polar coordinates you should have learned that dxdy becomes r dr dtheta.
(That is, of course, because the Jacobian is "r":
)