It appears to me the u and w substitutions won't do you much good (u might, but it seems a bit too complicated to try). The v substitution is the obvious one to work.

INT x*sqrt(x - 3) dx

Let v = x - 3 --> x = v + 3 <--> dx = dv

INT (v + 3)*sqrt(v) dv

= INT v*sqrt(v) + 3*sqrt(v) dv

= INT v^(3/2) + 3v^(1/2) dv

= 2/5*v^(5/2) + 2v^(3/2) + C

As for your second problem, x*(x-3) dx, no substitution is needed. Check to make sure this is the correct problem.