In the first why not simply rcos(t) +r sin(t) =9

then r= 9/[cos(t)+sin(t)] ? By the way there are pblms with your algebra --no need to square both sides

In the second your integral Calculates the area inside the outer loop

you have to subtract the area of the inner loop

2integral 1/2( 2pi/3 to pi) f(theta)^2(dtheta)