Let denote the height of the centre of the ball above the vertex of the cone, and let denote the radius of the ball, so that . The volume displaced is the volume of that part of the ball that lies below height h. I'll cheat by quoting the formula given here for the volume of a frustum of a sphere, which tells you that it is
To save writing, it's convenient to write for . So replace by in the above formula, and you get
To find the value of that maximises , differentiate (with respect to ) and put the derivative equal to 0. After some simplification you should find that the maximum occurs when , from which
. Finally, replace by and remember that .