The first step is to find the x and y coordinates of the corners for a given z. And to do that you need to work out a formula for the cone. Here' what I would do: Looking just at the first octant, a cross section at the x,z plane gives a line from x= 0, z= 1 to x= 2, z= 0. That is given by z= 1- x/2 or x= 1- 2z. Similarly, a cross section at the y,z plane gives a line from y= 0, z= 1 to y= 1, z= 0. That is given by z= 1- y or y= 1- z.

So at a given "z", the cross section is an ellipse with semi-axes 1- 2z and 1- z. The equation of that ellipse is .

Now we can say that we want to maximize the volume, V= xyz, subject to the constraint

To solve that using "Lagrange multipliers" set .

That, together with the constraint, give four equations for x, y, z, and .