I'm coming up with 4a squared - 12 ab + 9b still not sure if I'm getting it right though?
The final result will contain terms of order 3, so there will be terms containing the cube of $\displaystyle a$ and of $\displaystyle b$, as well as terms containing $\displaystyle a^2b$ and $\displaystyle ab^2$