My book is asking me to find this before the chapter is over, it is quite puzzling and one of the toughest questions at the end of the lesson.
i don't know if i can explain this without making it overly complicated, but i will try.
to understand what is going on, let's write (x',y') in POLAR coordinates as (rcos(ψ),rsin(ψ)), where ψ is the angle between the point (x',y') and the x'-axis.
(we don't need to know what r and ψ actually are for what we are going to do, but if you MUST know:
r = √(x'2+y'2)
ψ = arctan(y'/x'), if x' > 0
= π/2 if x' = 0 and y' > 0
= -π/2 if x' = 0 and y' < 0
= arctan(y'/x') + π, if x < 0, y ≥ 0
= arctan(y'/x') - π, if x < 0, y < 0 (and undefined for the origin) ...these rules are to make sure we get the angle ψ in the right "quadrant").
now, if we are given a point (x',y') in the "rotated plane", we might want to know what it's "normal coordinates are". since we got (x',y') after rotating through an angle θ, to recover the coordinates in the "un-rotated axes", we need to add in this angle of rotation:
(x,y) = (rcos(ψ+θ), rsin(ψ+θ)).
using the trigonometry angle-sum identities:
cos(ψ+θ) = cos(ψ)cos(θ) - sin(ψ)sin(θ)
sin(ψ+θ) = sin(ψ)cos(θ) + cos(ψ)sin(θ), we have:
(x,y' = (r(cos(ψ)cos(θ) - sin(ψ)sin(θ)), r(sin(ψ)cos(θ) + cos(ψ)sin(θ)))
= (rcos(ψ)cos(θ) - rsin(ψ)sin(θ), rsin(ψ)cos(θ) + rcos(ψ)sin(θ)).
but now x' = rcos(ψ), and y' = rsin(ψ), so we can just substitute back in:
(x,y) = (x'cos(θ) - y'sin(θ), y'cos(θ) + x'sin(θ))
if we want to run this procedure in reverse (and recover (x',y') from the xy-coordinates), we need to SUBTRACT the angle of rotation:
(and here, we write (x,y) = (rcos(φ), rsin(φ))
(x',y') = (rcos(φ-θ), rsin(φ-θ)) = (rcos(φ)cos(θ) + rsin(φ)sin(θ), rsin(φ)cos(θ) - rcos(φ)sin(θ))
= (xcos(θ) + ysin(θ), ycos(θ) +- xsin(θ)).
perhaps an example will make this clearer:
suppose we have x'y'-coordinates (rotated axes) of (1,0), and the axes have been rotated 1/8 of a turn (so θ = π/4).
then the xy-coordinates are:
x = 1*cos(π/4) - 0*sin(π/4) = √2/2 + 0 = √2/2
y = 0*cos(π/4) + 1*sin(π/4) = 0 + √2/2 = √2/2, so the xy-coordinates are (√2/2, √2/2) (which is exactly what you would expect the coordinates of a unit length at a 45 degree angle to be).
the main use of this is in conic sections, where you choose an angle of axis rotation θ to make an xy-term disappear, and thus be able to put a conic curve in "standard form" in x'y'-coordinates. this allows easy determination of the foci, and vertices (and/or asymptotes), which then can be translated "back" to xy-coordinates for sketching.