I was always taught a general transformation for a rotation:

Where is the number of degrees anticlockwise that the shape is being rotated. I don't understand your method, admittedly, because I never had to use it, but I would suspect that there is a mistake within your working because it contradicts with my method, which is a general formula. I'll check your method more carefully and see if I can work out where the error lies - it may be that for whatever reason my formula doesn't apply here.