The line is a straight line through the origin at angle to the x-axis.
One way of determining a matrix representing any linear transformation is to see what it does to the basis vectors.
To find the reflection of the point (1, 0) in the line (for simplicity, I am going to write ) we look at the line through (1, 0) perpendicular to the line y= mx.
The line through (1, 0) perpendicular to y= mx has slope -1/m and so equation y= -(1/m)(x- 1).
It also makes a right triangle with hypotenuse from (0, 0) to (1, 0) and angle and so the "opposite side", the length of the line from (1, 0) to the original line, of . The reflection of (1, 0) in the line y= mx, is at that same distance on the other side of the line. That is, it lies on the line y= -(1/m)(x- 1) and has distance from (1, 0) equal to . Using the distance formula we want .
Now, use the fact that . so that equation becomes so [tex]|x- 1|= 2sin^2(\alpha)[tex]. It is clear, geometrically, that x< 1 so |x- 1|= 1- x and we have and . From y= -(1/m)(x- 1), .
That is, the first reflection maps to and that means that the first column of the matrix corresponding to that reflection is .
There is probably some simpler formula to get that, but that is my take on the situation.
Do the same kind of analysis with (0, 1) to get the second column of the matrix.
Of course, the matrix representing the second reflection will be exactly the same thing with instead of . The matrix representing both reflections, one after the other, is the product of those two matrices (in the correct order).