cut four squares of size x by x from each corner, then fold up the flaps.

resulting 3D box dimensions are (3-2x) by (3-2x) by x

that's the difficult part ... do the calculus and find the value of x that will maximize the volume.

remember your usable domain ... 0 < x < 3