The force at D is in the opposite direction, so you should have:

M1 = r x F = (-3i + 2j) x (-400cosx i - 400sinx j) = (1200sinx + 800cosx) k

and so the total moment is

Meq = (1200sinx + 800cosx - 400cosx) k = 400(3sinx + cosx) k

(and you'll write instead of x in the solution, of course)

Part (d) should be easy - just substitute for x.

For (b), you can take the derivative and set it to zero. (I don't see how to do it without calculus - if you haven't had calculus, post again)

For (c), notice that if x is between 180 and 270 degrees, the forces will both be in the opposite direction, so the moment will have the opposite sign. So it must have passed through zero somewhere - just set Meq to zero and solve for x.

Post again if you're still having trouble.