Integration by parts.

$\displaystyle \int x^2 e^{-x} dx$

Let, $\displaystyle u=x^2$ and $\displaystyle v'=e^{-x}$

Thus,

$\displaystyle u'=2x$ and $\displaystyle v=-e^{-x}$

Thus,

$\displaystyle uv-\int u'vdx$

Thus, (note the signs change

)

$\displaystyle -x^2e^{-x}+2\int xe^{-x} dx$

Do the same integration by parts on this integral.

$\displaystyle u=x$ and $\displaystyle v'=e^{-x}$

Thus,

$\displaystyle u'=1$ and $\displaystyle v=-e^{-x}$

Thus, (note the parantheses

)

$\displaystyle -x^2e^{-x}+2\left( -xe^{-x}+\int e^{-x} dx\right)$

Thus,

$\displaystyle -x^2e^{-x}-2xe^{-x}-2e^{-x}+C$

Thus,

$\displaystyle -e^{-x}(x^2+2x+2)+C$