I have done the first part. I can't think of a way to do the second part other than by making guesses. The answers are y=2x and y=-0.5x.
Yes. To do the first part, look at $\displaystyle \begin{pmatrix}2 & 1 \\ 2 & -2\end{pmatrix}\begin{pmatrix}x \\ y\end{pmatrix}= \begin{pmatrix}2 & 1 \\ 2 & -2\end{pmatrix}\begin{pmatrix}x \\ 3x\end{pmatrix}$$\displaystyle = \begin{pmatrix}2x+ 3x \\ 2x- 6x\end{pmatrix}= \begin{pmatrix}5x \\ -4x\end{pmatrix}$.
So x'= 5x and y'= -4x. x= x'/5 so y'= -4(x'/5)= (-4/5)x'.
Any "line through the origin" is of the form y= ax with slope a. The line at right angles to that is y= (-1/a)x so you want to find a such that
$\displaystyle \begin{pmatrix}2 & 1 \\ 2 & -2\end{pmatrix}\begin{pmatrix}x \\ y\end{pmatrix}= \begin{pmatrix}2 & 1 \\ 2 & -2\end{pmatrix}\begin{pmatrix}x \\ ax\end{pmatrix}$$\displaystyle =\begin{pmatrix}2x+ ax \\ 2x- 2ax\end{pmatrix}= \begin{pmatrix}(2+a)x \\ (2- 2a)x\end{pmatrix}= \begin{pmatrix}x' \\ -(1/a)x'\end{pmatrix}$
That is, we must have x'= (2+a)x and y'= (-1/a)x'= (2- 2a)x. Dividing one equation by the other so that we eliminate x' on both sides of the equation, $\displaystyle -\frac{1}{a}= \frac{2+ a}{2+ 2a}$. Multiply on both sides by (2+ 2a)a to get rid of the fractions.
$\displaystyle 2+ 2a= (2+ a)a= 2a+ a^2$ which reduces to $\displaystyle a^2= 2$.
Ouch, I divided the two equations incorrectly. Dividing x'= (2+ a)x by (-1/a)x'= (2- 2a)x gives $\displaystyle \frac{x'}{-\frac{1}{a}x'}= \frac{(2+a)x}{(2- 2a)x}$ or $\displaystyle -a= \frac{2+a}{2- 2a}$. Multiplying both sides by 2- 2a gives -(2- 2a)a= -2a+ 2a^2= 2+ a. $\displaystyle 2a^2- 3a2 2= (2a+ 1)(a- 2)= 0$ which has roots a= 2 and a= -1/2. the lines, y= ax, are y= 2x and y= (-1/2)x as you said.