Hello walreinlim88I think there is a mistake in the question. I believe the transformation matrix should be:which represents a reflection in the line through the origin that makes an anticlockwise angle with the -axis, where .
See the diagram attached, where PQ is the mirror-line. Note that and .
It's fairly straighforward to show that if , then , and hence the equation of the mirror (invariant) line is(Use and .)
Can you continue from here?
Grandad