$\displaystyle \tan2x = 8 \cos^2x - \cot x [0,\frac{\pi}{2}]$
Arrange it all on one side so it's
$\displaystyle \tan 2x + \cot x - 8 \cos^2 x = 0$.
Now the strategy is to factor the left side to get smaller pieces that are forced to be equal to zero.
Start by recalling $\displaystyle \tan 2x = \frac{\sin 2x}{\cos 2x}$ and $\displaystyle \cot x = \frac{\cos x}{\sin x}$, so multiply through by $\displaystyle \cos 2x \sin x$.
You get:
$\displaystyle \sin 2x \sin x + \cos x \cos 2x - 8 \cos^2 x \cos 2x \sin x = 0$.
Now remember $\displaystyle \sin 2x = 2\sin x \cos x$. So:
$\displaystyle 2 \sin^2 x \cos x + \cos x \cos 2x - 8 \cos^2 x \cos 2x \sin x = 0$.
And now we can make our first factor, which is $\displaystyle \cos x$.
$\displaystyle \cos x \left(2 \sin^2 x + \cos 2x - 8 \cos x \cos 2x \sin x \right) = 0$.
No we know $\displaystyle \cos x = 0$ when $\displaystyle x = \frac{\pi}{2}$ (in your range), so that is one solution. Let's forget about that factor now and concentrate on the rest:
$\displaystyle 2 \sin^2 x + \cos 2x - 8 \cos x \cos 2x \sin x = 0$.
Recall that $\displaystyle \cos 2x = \cos^2 x - \sin^2 x$, and apply this to the middle term:
$\displaystyle 2 \sin^2 x + \cos^2 x - \sin^2 x - 8 \cos x \cos 2x \sin x = 0$
Simplifies to:
$\displaystyle \sin^2 x + \cos^2 x - 8 \cos x \cos 2x \sin x = 0$.
Now, remember that $\displaystyle \sin^2 x + \cos^2 x = 1$, so we now have:
$\displaystyle 1 - 8 \cos x \cos 2x \sin x = 0$.
But refer again to the fact that $\displaystyle 2 \sin x \cos x = \sin 2x$. so we have:
$\displaystyle 1 - 4 \sin 2x \cos 2x = 0$.
Apply the above identity again:
$\displaystyle 1 - 2 \sin 4x = 0$.
Now solve for $\displaystyle x$ by just isolating it the old fashioned way. From this, you should get two more values in the interval $\displaystyle 0$ to $\displaystyle \frac{\pi}{2}$, making three values total because of the one we found earlier.