It might be easier to use a Hyperbolic Substitution rather than a Trigonometric. Recall that $\displaystyle \displaystyle \cosh^2{(x)} - \sinh^2{(x)} = 1 \implies \cosh^2{(x)} - 1 = \sinh^2{(x)}$. Also $\displaystyle \displaystyle \frac{d}{dx} \left[ \cosh{(x)} \right] = \sinh{(x)}$.
So in this case, a substitution $\displaystyle \displaystyle x = \frac{5}{3}\cosh{(x)} \implies dx = \frac{5}{3}\sinh{(x)}\,dx$ is appropriate.
You should have 3x on the hypotenuse.
So your substitution is $\displaystyle \frac{5}{3x}=\sin{\theta}$ or $\displaystyle \frac{1}{x}=\frac{3}{5}\sin{\theta}$ and then $\displaystyle \cot{\theta}=\frac{\sqrt{9x^2-25}}{5}$ or $\displaystyle 5\cot{\theta}=\sqrt{9x^2-25}$.
- Hollywood