I am having some difficulty answer this type of question.

x^2-5x+y^2=3 rotated by pi/3

cos pi/3= 1/2 sin pi/3= sqrt{3}/2

But I am unsure how to proceed my book says

x with x cos theta + y sin theta

y with -x sin theta + y cos theta

Then plug this in the original equation

I think the graph is a circle