Firstly before you type in a formula, click the sigma button (second-top row, third from the right) and make sure the formula is between the two math brackets.

You'll want to check your algebra before proceeding...I'm not sure what you did but \(\displaystyle cot^2x = 1\) isn't the right direction. Trying factoring out the 8 to start.

Are you allowed to use a calculator or not?