Take the origin at a corner of the box with the axes along the edges.
Let r be the radius of a styrofoam sphere and R equal the radius of the bowling ball. The distance from the origin to the center of the bowling ball is equal to the sum of the distance from the origin to the center of the styrofoam sphere nearest the origin and the distance between the center of this sphere and the center of the bowling ball so:
Here is one way.
Take any octant of the cube/box whose side is 2R. This octant is a cube whose side is R, which we call octant R-cube, will contain an octant of the large sphere whose radius is R, and a whole small sphere whose radius is r.
A straight line from the center of the large sphere (call it point A), passing through the center of the small sphere (call it point B) will also pass through the corner (call it point C) of the octant R-cube diagonally from the center of the R-sphere. This diagonal from A, through B, to C is say, d long.
So, using the edges of the octant R-cube,
d^2 = R^2 +R^2 +R^2 = 3R^2
d = sqrt[3R^2]
d = R[sqrt(3)]................(1)
Also,
d = R +2r +x .............(i)
where x = distance from nearest edge of the small sphere (call it point E) to the corner C. Point E is on the line AC. And EC = x.
Repeat the scheme above on the small octant r-cube that contains the point/corner C also.
(r+x)^2 = r^2 +r^2 +r^2 = 3r^2
r +x = r[sqrt(3)]
x = r[sqrt(3)] -r
x = r[sqrt(3) -1] ............**
Substitute that into (i),
d = R +2r + r[sqrt(3) -1]
d = R +r[2 +sqrt(3) -1]
d = R +r[sqrt(3) +1] ...............(2)
Then, d from (1) equals d from (2),
R[sqrt(3)] = R +r[sqrt(3) +1]
Isolate r,
R[sqrt(3)] -R = r[sqrt(3) +1]
R[sqrt(3) -1] = r[sqrt(3) +1]
r = R[sqrt(3) -1] / [sqrt(3) +1]
r = R{[sqrt(3) -1] / [sqrt(3) +1]}
Rationalize the denominator of the fraction in the righthand side.
Multiply both numerator and denominator by the conjugate of the denominator, which is [sqrt(3) -1],
r = R{[3 -2sqrt(3) +1] / [3 -1]}
r = R{[4 -2sqrt(3)] / [2]}
r = R[2 -sqrt(3)] -----------------------answer.