i ended up with (x - 2)(x + 1) but that is completely off and i dont know where i went wrong...
Your major mistake is the way you go about factoring the second term. Instead of using the times sign x I'm just going to write it like this:
$\displaystyle \frac{(x^2-2x-3)(x^2+2x)}{(x^2-x-6)(x^2-4x)}$
It's the same thing. You factored it all wrong, can you see where the factoring went wrong?
Just re-factor these one-by-one,
$\displaystyle (x^2-2x-3)$
$\displaystyle (x^2+2x)$
$\displaystyle (x^2-x-6)$
$\displaystyle (x^2-4x)$
Edit: I 'd like to add that all of the errors you make (that (I've seen so far) are just factoring quadratic expressions, so you should review that. You seem to know how to approach rational expressions like this methodologically (factoring, cancelling, and simplifying,) but you're hung-up on the factoring bit.
It looks like you've concluded that $\displaystyle (x^2+2x)=(x+2)(x+2) $ which isn't true. $\displaystyle (x+2)(x+2)=x^2+4x+4$. When factoring just try to expand them out and ask yourself if they make any sense.
$\displaystyle (x^2-4x)=x(x-4)$