A little help:
Por every $\displaystyle (x,y)\in \mathbb{R}^2$ find the symmetric point $\displaystyle (x',y')$ of $\displaystyle (x.y)$ with respect to $\displaystyle Y=mX$ . You'll obtain an expression of the form:
$\displaystyle \begin{bmatrix}{x'}\\{y'}\end{bmatrix}=A\begin{bma trix}{x}\\{y}\end{bmatrix}$
Fernando Revilla
To find the matrix corresponding to a linear transformation from $\displaystyle R^2$ to $\displaystyle R^2$, it is enough to see what it does to the "basis vectors", <1, 0> and <0, 1>.
Notice that $\displaystyle \begin{bmatrix}a_{11} & a_{12} \\ a_{21} & a_{22}\end{bmatrix}\begin{bmatrix}1 \\ 0\end{bmatrix}= \begin{bmatrix}a_{11} \\ a_{21}\end{bmatrix}$
and than
$\displaystyle \begin{bmatrix}a_{11} & a_{12} \\ a_{21} & a_{22}\end{bmatrix}\begin{bmatrix}0 \\ 1\end{bmatrix}= \begin{bmatrix}a_{12} \\ a_{22}\end{bmatrix}$
so what the linear transformation does to <1, 0> and <0, 1> gives you the two columns of the matrix.
Now, the line, through the origin with slope m, y= mx, contains the points (0, 0) and (1, m) and so is in the direction of the vector <1, m>. To find the "reflection" of <1, 0> in that line, find its projection onto the vector <1, m>, then subtract that from <1, 0> to find its component perpendicular to the line. Subtract twice that vector from <1, 0> (subtracting once puts you on the line, subtracting again, on the other side) to find the reflection in the line.
Alternative 3 :
Some courses of Linear Algebra cover the following and well known theorem:
If $\displaystyle H:a_1x_1+\ldots a_nx_n=0$ is a hyperplane of $\displaystyle \mathbb{R}^n$ , then the projection matrix on $\displaystyle H$ with the usual inner product is:
$\displaystyle P=I-\dfrac{uu^t}{u^tu},\quad u=(a_1,\ldots,a_n)^t$
As a consequence the simmetry matrix on $\displaystyle H$ is
$\displaystyle S=2P-I=I-2\dfrac{uu^t}{u^tu}$
In our case, $\displaystyle u=(m,-1)^t$ so,
$\displaystyle S=\begin{bmatrix}{1}&{0}\\{0}&{1}\end{bmatrix}-\dfrac{2}{m^2+1}\begin{bmatrix}{m}\\{-1}\end{bmatrix}\begin{bmatrix}{m}&{-1}\end{bmatrix}=\dfrac{1}{m^2+1}\begin{bmatrix}{1-m^2}&{2m}\\{2m}&{m^2-1}\end{bmatrix}
$
Fernando Revilla