I've attached a sketch of the gutter.

1. The area A of the cross section is calculated by:

A = x * y

2. You already know that x + y + x = 12 ==> y = 12 - 2x

3. Plug in this term for y into the first equation and you'll get a function with respect to x:

A(x) = x * (12 - 2x) = 12x - 2x²

4. Now use the first derivative to get the value of x which yields an extreme area:

I've got x = 3. Therefore y = 6 and the maxiumum area is A_{max} = 18