A conical glass of depth h and generating angle A, there is carefully dropped spherical iron ball of such size as to cause the greatest overflow.
Show that the radius of the sphere is equal to (h sin A)/(sin A + cos 2A).
Let denote the height of the centre of the ball above the vertex of the cone, and let denote the radius of the ball, so that . The volume displaced is the volume of that part of the ball that lies below height h. I'll cheat by quoting the formula given here for the volume of a frustum of a sphere, which tells you that it is
To save writing, it's convenient to write for . So replace by in the above formula, and you get
To find the value of that maximises , differentiate (with respect to ) and put the derivative equal to 0. After some simplification you should find that the maximum occurs when , from which
. Finally, replace by and remember that .