Let be the centers of the circles. (The bigger circle has the center ),
In the right triangle CAB apply Pitagora:
Now solve the quadratic. Remember that
Ok, here's what I think.
In my diagram I have drawn a line from the center of the larger gear to the tangent point of the smaller gear.
Use the Pythagorean Theorem to find its length.
Find angle DBA using Arctan.
Angle BAC is also 59.9314 since alternate interior angles are congruent (BD and CA are parallel because we have two lines perpendicular to the same line)
Now look at triangle ABC. We know , a = 20, and angle BAC = 59.9314.
Use the Law of Cosines.
This all boils down to
Apply the quadratic formula to get your 2 results. One you have to throw away because it is bigger than the larger gear radius.
Now you have your answer.