How do I find the matrix A to the linear transformation T: R^3 --> R^3
it's defined by;
1. reflection against 3x - 6y + 5z = 0
then
2. projection onto 2x + 6y + 4z = 0
Here are the formulas that you need (I won't do the actual question for you).
Suppose that $\displaystyle \mathbf{n}=(a,b,c)$ is a unit vector (so that $\displaystyle a^2+b^2+c^2=1$). Then the projection onto the one-dimensional subspace spanned by n is $\displaystyle P_{\mathbf{n}} = \begin{bmatrix}a^2&ab&ac\\ab&b^2&bc\\ac&bc&c^2\end {bmatrix}$.
If px + qy + rz = 0 is the equation of a plane, let n be a unit vector orthogonal to the plane. So $\displaystyle \textstyle\mathbf{n} = \frac1{\sqrt{p^2+q^2+r^2}}(p,q,r)$. Then the matrix of the projection onto the plane is $\displaystyle I-P_{\mathbf{n}}$, and the matrix of the reflection in the plane is $\displaystyle I-2P_{\mathbf{n}}$.
To find the matrix for the composition of two such operations, form the matrices for each operation, then multiply them. So the matrix for reflection in 3x - 6y + 5z = 0 followed by projection onto 2x + 6y + 4z = 0 is $\displaystyle (I-P_{\mathbf{n}})(I-2P_{\mathbf{m}})$, where m and n are the normalised versions of (3,-6,5) and (2,6,4) respectively.
Unless I've also messed it up, you should get
$\displaystyle \textstyle\mathbf{m} = \frac1{\sqrt{70}}(3,-6,5),\quad \mathbf{n} = \frac1{\sqrt{56}}(2,6,4) = \frac1{\sqrt{14}}(1,3,2)$,
$\displaystyle P_\mathbf{m} = \frac1{70}\begin{bmatrix}9&-18&15\\ -18&36&-30\\ 15&-30&25\end{bmatrix},\qquad P_\mathbf{n} = \frac1{14}\begin{bmatrix}1&3&2\\ 3&9&6\\ 2&6&4\end{bmatrix}$,
$\displaystyle I-2P_\mathbf{m} = \frac1{35}\begin{bmatrix}26&18&-15\\ 18&-1&30\\ -15&30&10\end{bmatrix},\qquad I-P_\mathbf{n} = \frac1{14}\begin{bmatrix}1&3&2\\ 3&9&6\\ 2&6&4\end{bmatrix}$.
So the answer should be $\displaystyle \frac1{35\times14}\begin{bmatrix}26&18&-15\\ 18&-1&30\\ -15&30&10\end{bmatrix}\begin{bmatrix}1&3&2\\ 3&9&6\\ 2&6&4\end{bmatrix}$ (I'm not prepared to do the arithmetic to evaluate that).