A conical glass of depth h and generating angle A, there is carefully dropped spherical iron ball of such size as to cause the greatest overflow.
Show that the radius of the sphere is equal to (h sin A)/(sin A + cos 2A).
Let $\displaystyle x$ denote the height of the centre of the ball above the vertex of the cone, and let $\displaystyle r$ denote the radius of the ball, so that $\displaystyle r = x\sin A$. The volume displaced is the volume of that part of the ball that lies below height h. I'll cheat by quoting the formula given here for the volume of a frustum of a sphere, which tells you that it is $\displaystyle V = \tfrac{\pi}3(r+h-x)^2(2r-h+x).$
To save writing, it's convenient to write $\displaystyle s$ for $\displaystyle \sin A$. So replace $\displaystyle r$ by $\displaystyle xs$ in the above formula, and you get $\displaystyle V = \tfrac{\pi}3\bigl(h-x(1-s)\bigr)^2\bigl(x(2s+1)-h\bigr).$
To find the value of $\displaystyle x$ that maximises $\displaystyle V$, differentiate (with respect to $\displaystyle x$) and put the derivative equal to 0. After some simplification you should find that the maximum occurs when $\displaystyle x = \frac h{s+1-2s^2}$, from which
$\displaystyle r = xs = \frac {hs}{s+1-2s^2}$. Finally, replace $\displaystyle s$ by $\displaystyle \sin A$ and remember that $\displaystyle \cos(2A) = 1-2\sin^2A$.