Let's call (theta) = t for convenience.

sin(3t) = sin(2t + t) = sin(2t)cos(t) + sin(t)cos(2t)

= 2sin(t)cos(t)*cos(t) + sin(t)*(1 - 2sin^2(t))

= 2sin(t)cos^2(t) + sin(t) - 2sin^3(t)

= 2sin(t)(1 - sin^2(t)) + sin(t) - 2sin^3(t)

= 2sin(t) - 2sin^3(t) + sin(t) - 2sin^3(t)

= 3sin(t) - 4sin^3(t)

So your polar equation is:

r = 6sin(t) - 8sin^3(t)

Now,

r = (x^2 + y^2)^{1/2}

t = atn(y/x)

This is probably not the easiest way to go. I mean, sin(atn(y/x))? It can be done, but we have to do this for four different quadrants. Let's try this:

y = r*sin(t)

sin(t) = y/r = y/(x^2 + y^2)^{1/2}

So...

(x^2 + y^2)^{1/2} = 6*y/(x^2 + y^2)^{1/2} - 8*y^3/(x^2 + y^2)^{3/2}

I'm going to clear the fractions and multiply both sides by (x^2 + y^2)^{3/2}:

(x^2 + y^2)^2 = 6y(x^2 + y^2) - 8y^3

And you can simplify from here as you like. It's not going to simplify into anything pretty anyway.

-Dan