such that when the rotation is applied the coefficient of x'y' is zero. Then you can apply the discriminant test to the equation in x' and y' to determine the kind of curve.
For example, in this case I get the coefficient of the x'y' term to be
The solution here is nasty and I could only do it numerically. I got . The equation becomes
which is clearly an ellipse.