{5xy(x² + y²) = -160
{x² + 5xy +y² = -12
???
The first thing I noticed is that you recognize two equal parts in both equations, the $\displaystyle 5xy$ part and $\displaystyle x^2+y^2$
So you can 'simplify' the system by saying, let $\displaystyle x^2+y^2=a$ and $\displaystyle 5xy=b$ therefore you get the two simultaneous equations:
$\displaystyle b\cdot a=-160$ (1)
$\displaystyle a+b=-12$ (2)
Solve this system and afterwards do the back-substitution.
You're welcome!
I'll give you a part of the solution,
Solving the system by reforming (1) to $\displaystyle b$ and substituting this in (2):
$\displaystyle a-\frac{160}{a}=-12$
$\displaystyle \Leftrightarrow a^2-160=-12a$
$\displaystyle \Leftrightarrow a^2+12a-160=0$
Two solutions: $\displaystyle a=8$ or $\displaystyle a=-20$
But we have to reject $\displaystyle a=-20$ because $\displaystyle a=x^2+y^2$ is always >0.
If $\displaystyle a=8$ then $\displaystyle b=-20$
So now you can do the back-substitution and solve it.