Found the answer.
Wasted so much time typing =.=""
In my second equation
What I did isA particle P of mass 8 kg rests on a smooth plane inclined at 30 degrees to the horizontal. It is connected by a light inextensible string passing over a smooth light pulley at the top of the plane, to a particle of mass 3kg which is on a smooth plane inclined at 60 degrees to the horizontal.
Find the acceleration of the system and the tension in the string.
Used the calculator for simultaneous equation, which is not the answer in the back of the book. It should be 1.249m/s^2 and 29.21N. But i've got -64.25 and -12.9
many thanks for your help!