The error is in the second to the last line: you didn't distribute the x^2.

How about we write it this way: dy/dx = y' It works out to be a bit neater.

Then we have

y = [x^2]*[ln(x)cos(3x)]

y' = [x^2]' * [ln(x)cos(3x)] + [x^2] * [ln(x)cos(3x)]'

Now

[x^2]' = 2x

and

[ln(x)cos(3x)]' = [ln(x)]' * [cos(3x)] + [ln(x)] * [cos(3x)]'

where

[ln(x)]' = 1/x and [cos(3x)] = -sin(3x) * 3 <-- By the chain rule.

So

[ln(x)cos(3x)]' = (1/x)*cos(3x) - 3*ln(x)*sin(3x)

Thus

y' = [x^2]' * [ln(x)cos(3x)] + [x^2] * [ln(x)cos(3x)]'

y' = 2x*ln(x)*cos(3x) + x^2*[(1/x)*cos(3x) - 3*ln(x)*sin(3x)]

y' = 2x*ln(x)*cos(3x) + x*cos(3x) - 3x^2*ln(x)*sin(3x)

-Dan