Hi. I need a hand with this. I tried to solve it in different ways and didn't manage to get to the book answer.
Try it this way:
$\displaystyle 10x^4y^2+20x^3y^3-30x^2y^4$
Find GCF to be $\displaystyle 10x^2y^2$
Now factor out the GCF:
$\displaystyle 10x^2y^2(x^2+2xy-3y^2)$
Factor the trinomial:
$\displaystyle 10x^2y^2(x+3y)(x-y)$
To answer your 1st question, if you factor a 2 out of your last step, you will arrive at the correct solution.
In your second attempt, you need to factor out a $\displaystyle 10x^2y^2$ to arrive at the same conclusion.