Are you sure about that domain. I might be missing something in your explanation of the problem (or I might have forgotten something from Calc 3), but shouldn't it be a closed domain? b^2 < 2a is open and infinite, and so the integration would be undefined.

I'll let you work out the domain, I'll help setup the integration:

DBL INT {over D} 1/(1 + a^2 + b^2) da db

You are right in that we need to convert to polar coordinates (I'll let "O" represent "theta").

Let b = rcosO

Let a = rsinO

Let da db = dA = r dr dO ... I hope you know how I got this

DBL INT {over D} 1/(1 + r^2sin^2(O) + r^2cos^2(O)) r dr dO

DBL INT {over D} 1/(1 + r^2) r dr dO

Let 1 + r^2 = u <--> 2r dr = du --> r dr = 1/2 du

DBL INT {over D} 1/(2u) du dO

I can go no further because over the domain, D, r is a function of O. In order to integrate 1/(2u) du, we need to know the limits of r. Once you do know the limits, this integration becomes easy. After integrating r, you will be left with the integration of a function for O that is pretty straight forward, I think.

INT {limits of O} f(O) dO, where f(O) is most likely some trig function.