Im so rusty... im sure I should be looking at some bifurcation stuff but I do this by handwaving... and it might be wrong!

I assume that it'll be a regular tetrahedron that maximizes volume.

it can be shown that the angle between the face and an edge of a regular tetrahedron is arctan(2^(1/2))

Im just going to consider a slice thru the middle of the sphere to make it a 2d problem. You can align it so this slice is a plane passing through two of the verticies.

Consider a circle centered at (0, R) with radius R. Then the top of the circle is at (0, 2R). The edge of the tetrahedron represents a line passing through (0, 2R) with a slope of 2^(1/2). It can be shown that the equation of this line turns out to be y = (2^(1/2))x +2R.

the equation of the circle is x^2 + (y-R)^2 = R^2

substitute for y here with our line to find the intersection points. The unknown one turns out to be (x,y) = (-2^(3/2)*R/3, (2/3)*R)

Then a = side length of tetrahedron = ((0 + 2^(3/2)*R/3)^2+(2*R - (2/3)*R)^2)^(1/2) = 2*(2/3)^(1/2)*R

--> volume of tetrahedron = (8*R^3)/(9*3^(1/2))

I might have messed up some algebra somewhere so you should check it yourself. Unfortunately I dont use any calculus here either... prolly would have been easier.