"u= 3theta" isn't going to help you with "2theta". Instead use trig identities to reduce sin(3theta) and cos(2theta) to sin(theta) and cos(\theta)

cos(2theta)= cos^2(theta)- sin^2(theta) you should know "by heart".

sin(3theta) is more complicated. sin(3theta)= sin(2theta+ theta)= sin(2theta)cos(theta)+ cos(2theta)sin(theta)= (2cos(theta)sin(theta))cos(theta)+ (cos^2(theta)- sin^2(theta))sin(theta)= 3cos^2(theta)sin(theta)- sin^2(theta)

(If that is really "sin(3theta^2)" rather than "sin^2(3theta)", that integral cannot be done in terms of elementary functions.)