since the sheet is square you should cut a squares from the corners not rectangles because if you cut rectangle the hight of the box will be change from side to side

this is my answer

let the square side be x so the volume of the box equal

(12-2x)(12-2x)x=v

v=4x^3-48x^2+144x

dv/dx=12x^2+-96x+144

0=12(x^2-8x+12)

12(x-2)(x-6)=0

x=2 or x=6

substitute 2 and 6

v(6)=0

v(2)=128