Did you mean "fractions" for you title? This problem has nothing to do with "fractals".
Assuming that by "10% smaller", you mean each ball's diameter is 10% smaller than the next larger ball, or 90% of the nest larger ball's diameter, then you have a geometric progression- each ball's diameter is 1/.9= 1.11 times as large as the previous ball. The first ball has diameter 5 cm so the second has diameter 5(1.11)= 5.55 cm, the third has diameter , etc. If there are n balls, then the largest has diameter . Find integer n so that is just less than 12 cm.