Hello walreinlim88I think there is a mistake in the question. I believe the transformation matrix should be:which represents a reflection in the line through the origin that makes an anticlockwise angle with the -axis, where .

See the diagram attached, where PQ is the mirror-line. Note that and .

It's fairly straighforward to show that if , then , and hence the equation of the mirror (invariant) line is(Use and .)

Can you continue from here?

Grandad