from the first equation ...
sub for x in the second equation ...
expand, collect like terms, and solve the quadratic for y ... then go back and evaluate for x. note that x > 0.
I tried to simplify it down and go to this point:
I don't know if I did it right. From there, I have no idea what to do T_T The problem asks for a exact answer so everything must stay in radicals. I also don't know how to solve it using matrices...