Given A(5,2), B(-3,6), and C(1,-3), determine if triangle ABC is a right triangle.

So first I took a look at the angle between A and B:

cos theta = [5,2][-3,6] / magnitude of [5,2] x magnitude of [-3,6]

= -15 + 12 / square root of 25+4 x square root of 9+36

= -3 / square root of 1305

theta = about 85.2 degrees

Then I took a look at the angle between B and C:

cos theta = [-3,6][1,-3] / magnitude of [-3,6] x magnitude of [1,-3]

= -3 - 18 / square root of 9+36 x square root of 1+9

= -21 / square root of 450

theta = about 171.9 degrees

I didn't even calculate the angle between B and C, and i'm already over 180 degrees so where did I go wrong?