Hello, aeroflix!
$\displaystyle (3)\;\;\cot^{\text{-}1}(2x) - \tan^{\text{-}1}(x) \:=\:30^o$
Note that: .$\displaystyle \cot^{\text{-}1}(2x) \;=\;\tan^{\text{-}1}\!\left(\frac{1}{2x}\right)$ . . . (Think about it!)
So we have: .$\displaystyle \tan^{\text{-}1}\!\left(\frac{1}{2x}\right) - \tan^{\text{-}1}(x) \:=\:30^o$
Take the tangent of both sides:
. . $\displaystyle \tan\bigg[\tan^{\text{-}1}\!\left(\frac{1}{2x}\right) - \tan^{\text{-}1}(x)\bigg] \;=\;\tan30^o$
. . $\displaystyle \dfrac{\tan\left[\tan^{\text{-}1}(\frac{1}{2x})\right] - \tan\left[\tan^{\text{-}1}(x)\right]} {1 + \tan\left[\tan^{\text{-}1}(\frac{1}{2x})\right] \tan\left[\tan^{\text{-}1}(x)\right]} \;=\;\dfrac{1}{\sqrt{3}}$
. . $\displaystyle \dfrac{\frac{1}{2x} - x}{1 - (\frac{1}{2x})(x)} \;=\;\dfrac{1}{\sqrt{3}} \quad\Rightarrow\quad 2\sqrt{3}\,x^2 + 3x - \sqrt{3} \:=\:0$
Quadratic Formula: .$\displaystyle x \;=\;\dfrac{-3 \pm\sqrt{33}}{4\sqrt{3}} $
Therefore: . $\displaystyle x \;=\;\begin{Bmatrix}\;\;0.396143496 \\ \text{-}1.262168899 \end{Bmatrix}$
You're trying too hard. Recall that $\displaystyle \cot(x) = \frac{1}{\tan(x)}$
Thus $\displaystyle \tan(x) = \frac{1}{\tan(x)}$
Multiply by $\displaystyle \tan(x)$
$\displaystyle \tan^2(x) = 1$ which is equal to $\displaystyle \tan^2(x)-1^2 = 0$
Solve the quadratic, I would use the difference of two squares to solve
Edit: Oops, I posted the answer to B not A which is in the quote
From what I can tell you're right (although confusing lol) up to $\displaystyle \sec^2(x) =2$. However, when you took the square root you didn't take the negative.
$\displaystyle \sec(x) = \pm \sqrt{2}$
Taking the reciprocal of both sides: $\displaystyle \cos(x) = \pm \frac{1}{\sqrt{2}}$
I got answers of $\displaystyle \frac{\pi}{4}$ and $\displaystyle -\frac{\pi}{4}$ (or $\displaystyle \pm 45^{\circ}$ ) as solutions (although only principal solutions because no range is defined in the question)
$\displaystyle \sec(x) \neq 0$ so discard that solution (from factoring out $\displaystyle \sec(x)$)
From $\displaystyle 2\sec{x} + \sec^2{x} = \sec^2{x} + \sec^3{x}$
$\displaystyle 2\sec{x} = \sec^3{x}$
$\displaystyle 0 = \sec^3{x} - 2\sec{x}$
$\displaystyle 0 = \sec{x}(\sec^2{x} - 2)$
$\displaystyle 0 = \sec{x}(\tan^2{x} - 1)$ since $\displaystyle \tan^2{x} = \sec^2{x} - 1$
$\displaystyle \sec{x} = 0$ or $\displaystyle \tan^2{x} - 1 = 0$
Case 1: Does $\displaystyle \sec{x}$ ever equal $\displaystyle 0$?
Case 2: $\displaystyle \tan^2{x} - 1 = 0$
$\displaystyle \tan^2{x} = 1$
$\displaystyle \tan{x} = \pm 1$
$\displaystyle x = \frac{\pi}{4} + \frac{n\pi}{2}$ where $\displaystyle n \in \mathbf{Z}$
$\displaystyle x = \frac{\pi + 2n\pi}{4}$ where $\displaystyle n \in \mathbf{Z}$
$\displaystyle x = \frac{(2n + 1)\pi}{4}$ where $\displaystyle n \in \mathbf{Z}$.