I'm assuming you meant f(g(x)) = (x+5)/x

If f(x) = 1/x and g(x) = x+5 then f(g(x)) = 1/(x+5).

You're asking for how they got their answer, but try to see why their answer is right. Maybe you'll see it better this way. Let g(x) = y. Then f(g(x)) = f(y) = y/(y-5) = (x+5)/(x+5-5) = (x+5)/x.

I wouldn't use a systematic way to find f(x), just look at it and recognize that the numerator is g(x) and the denominator is g(x) - 5.