Here's how to start the calculation:
. (You'll notice that I'm being a bit vague about the sums. They go from 0 to m or n for the cosines, and from 1 to m or n for the sines. It gets messy if you write this all out really carefully.) Now multiply out those parentheses and notice that a lot of the terms disappear or simplify when you integrate them from –L to L. For example, is zero unless m=n, in which case it is 1. Also, , with a similar result for and .
When you have finished that calculation, you should be able to write it in the form
That is obviously minimised by taking and for each n.