One question on my assignment has me stumped. I've tried a number of different methods, but every time, I either get stuck somewhere, or the equation quickly gets ridiculously out of hand. I'm not sure where to start here.

I've used google and some math help sites to try to point myself in the right direction, but it hasn't been much help. I'm getting answers like this;

Which both seem very wrong, and don't help at all.

Can someone give me some advice?