the key to these types of problems is to identify a "control variable"... something you have control over, that you can vary, and that will determine the value of the thing you are trying to maximize.

In this case I chose a certain angle to be my control variable. Draw an upside down cone, and let alpha be the angle made between the side of the cone and the vertex (the straight up and down line from the top of the cone to the bottom). Then alpha can vary anywhere between Pi/2 (if it is lying perfectly flat) and zero (if it wrapped around itself completely, and standing straight up). Our job then is to find the alpha that maximizes the volume of the cone.

So our next step is to write the volume of the cone as a function of alpha. I couldn't remember any of the formulas of the volume of the cone, so I just figure it out using integration. Let r0 be the radius of the base of the cone, and h be the height of the cone. Then verify that volume can be given as:

V = (Pi/3)*h^3*tan(alpha)^2

But since h = r0 / tan(alpha),

V = (Pi/3)*r0^3 / tan(alpha)

Finally, what is r0 in terms of alpha? Let R be the radius of the circle you started with (which in your case is 4 inches). Then notice that the side of the cone will also be of length R. Then we have also that r0 = sin(alpha)*R.

Thus:

V = (Pi/3)*(sin(alpha)*R)^3 / tan(alpha)

= (Pi/3)*R^3*cos(alpha)*sin(alpha)^2

Thus we have volume as a function of alpha. Now all thats left is to find the maximum of this function as alpha varies from 0 to Pi/2. Differentiate:

V' = -1/3*Pi*R^3*sin(alpha)^3+2/3*Pi*R^3*cos(alpha)^2*sin(alpha)

Setting V' = 0, we find three solutions, corresponding to where sin(alpha) = 0, and when tan(alpha) = +/- sqrt(2), i.e.,

alpha = 0, arctan(sqrt(2)), -arctan(sqrt(2))

only one of these, arctan(sqrt(2)) sits between 0 and Pi/2. Verify by the second derivative test that this is indeed a maximum of the function V. Thus our maximum possible volume is:

V (R = 4, alpha = arctan(sqrt(2)) ) = 128*Pi*sqrt(3)/27