okay! start from the identity $\displaystyle \cot^{-1}(cx) = \int_c^{\infty} \frac{x}{x^2t^2 + 1} \ dt, \ x > 0, \ c > 0.$ therefore:
$\displaystyle \int_0^{\infty} \cot^{-1}(ax) \cot^{-1}(bx) \ dx = \int_0^{\infty} \int_a^{\infty} \int_b^{\infty} \frac{x^2}{(x^2y^2 + 1)(x^2z^2 + 1)} \ dy \ dz \ dx$
$\displaystyle =\int_a^{\infty} \int_b^{\infty} \int_0^{\infty} \frac{x^2}{(x^2y^2 + 1)(x^2z^2 + 1)} \ dx \ dy \ dz=\frac{\pi}{2} \int_a^{\infty} \int_b^{\infty} \frac{1}{yz(y+z)} \ dy \ dz$
$\displaystyle =\frac{\pi}{2} \int_a^{\infty} \frac{\ln(z+b) - \ln b}{z^2} \ dz = \frac{\pi}{2}\left[\frac{1}{a} \ln \left(\frac{a+b}{b} \right) + \frac{1}{b} \ln \left(\frac{a+b}{a} \right) \right].$
in the last integral i used integration by parts: $\displaystyle \ln(z+b) - \ln b = u, \ \ \frac{1}{z^2} dz = dv.$
Here is another method to solve the integral.
$\displaystyle
F(a,b)\;=\;\int_0^\infty \cot^{-1}(ax) \cot^{-1}(bx)\; dx
$
$\displaystyle
\frac{\partial^2 F}{\partial a \; \partial b}\;=\;\int_0^\infty \frac{x^2}{(a^2x^2+1)(b^2x^2+1)} \; dx\;=\frac{\pi}{2} \frac{1}{ab(a+b)}
$
$\displaystyle
\frac{\partial F}{\partial a}\;=\; \frac{\pi}{2a} \int_\infty^b \frac{1}{b(a+b)}\;db \;=\; \frac{\pi}{2a^2}\ln\frac{a}{a+b}
$
$\displaystyle
F(a,b)\;=\; \; \frac{\pi}{2} \int_\infty^a \frac{1}{a^2}\ln\frac{a}{a+b}\, da\;=\;\frac{\pi}{2}\left[\frac{1}{a} \ln \left(\frac{a+b}{b} \right) + \frac{1}{b} \ln \left(\frac{a+b}{a} \right) \right]
$
We integrate from $\displaystyle \infty$ because when the integral is 0 when either $\displaystyle a\text{ or } b \text{ is } \infty$
You are talking about the integration procedure, rather than the method itself. Again, the result turns out to be very simple or even simpler in terms of mathematical explanation. The method is correct and simpler anyway (at least in my view, Hahahaha...). Different from you, I like this method better. You may have a different view, that's fine.
I finally got some time to put other a detailed solution and here is the entire procedure:
$\displaystyle
F(a,b)\;=\;\int_0^\infty \cot^{-1}(ax) \cot^{-1}(bx)\; dx
$
$\displaystyle
\frac{\partial^2 F}{\partial a \; \partial b}\;=\;\int_0^\infty \frac{x^2}{(a^2x^2+1)(b^2x^2+1)} \; dx\;=\frac{\pi}{2}\; \frac{1}{ab(a+b)}
$
$\displaystyle
\frac{\partial F}{\partial a}\;=\; \frac{\pi}{2a} \int_\infty^b \frac{1}{b(a+b)}\;db \;=\; \frac{\pi}{2a^2}\ln\frac{a}{a+b}
$
$\displaystyle
F(a,b)\;=\; \; \frac{\pi}{2} \int_\infty^a \frac{1}{a^2}\ln\frac{a}{a+b}\, da\;=\;\frac{\pi}{2}\left[\frac{1}{a} \ln \left(\frac{a+b}{b} \right) + \frac{1}{b} \ln \left(\frac{a+b}{a} \right) \right]
$
We integrate from $\displaystyle \infty$ because when the integral is 0 when either $\displaystyle a\text{ or } b \text{ is } \infty$
Is this method OK?
we know $\displaystyle \int_0^{\infty} e^{-st} t^{n}~dt = \frac{\Gamma(n+1)}{s^{n+1}}$
so the integral equals to $\displaystyle \frac{ \Gamma(n) }{ a^n} - \frac{\Gamma(n)}{ b^n} $ $\displaystyle n \to 0 $
$\displaystyle = \lim_{n\to 0} \frac{ b^n - a^n}{n} $