Seems like the quadratic formula but I dont understand the steps
... the equation is quadratic in $\omega^2$ ...
$a = m^2$, $b = -4km$, $c = 3k^2$
$\omega^2 = \dfrac{-b \pm \sqrt{b^2-4ac}}{2a} = \dfrac{4km \pm \sqrt{(-4km)^2 - 4(m^2)(3k^2)}}{2(m^2)}$
$\omega^2 = \dfrac{4km \pm \sqrt{16k^2m^2 - 12k^2m^2}}{2(m^2)}$
$\omega^2 = \dfrac{4km \pm 2km}{2(m^2)}$
$\omega_1 = \sqrt{\dfrac{k}{m}}$
$\omega_2 = \sqrt{\dfrac{3k}{m}}$
Strictly speaking a quartic (fourth degree) equation will have four solutions. The other two would be $\displaystyle -\omega_1$ and $\displaystyle -\omega_2$. But a "frequency" is always a positive number.
Actually, I would say that the equation is quadratic in $m \omega^2.$
We start with $m^2 \omega^4 - 4km \omega^2 + 3k^2 = 0.$
$\text {Let } u = m \omega^2 \implies u^2 = m^2 \omega^4 \implies u^2 - 4uk + 3k^2 = 0 \implies$
$(u - 3k)(u - k) = 0 \implies u = 3k \text { or } u = k \implies $
$m \omega^2 = 3k \text { or } m \omega^2 = k \implies \omega = \pm \sqrt{\dfrac{3k}{m}} \text { or } \omega = \pm \sqrt{\dfrac{k}{m}}.$
The mechanics are easy. What may be hard is to see that the substitution
$u = m \omega^2$ will turn the whole thing into a simple quadratic.
EDIT: Halls points out that a quartic may have four different roots as this shows. But he also says that the negative solutions can be ignored.
The given equation is $\displaystyle m^2\omega^2- 4km\omega^2+ 3k^2= 0$. I would substitute x for $\displaystyle \omega^2$ (JeffM would use $\displaystyle x= m\omega^2$- either works). In terms of x, the equation becomes $\displaystyle m^2x^2- 4kmx+ 3k^2= 0$, a quadratic equation in $\displaystyle x$. By the quadratic formula, $\displaystyle x= \frac{4km\pm \sqrt{16k^2m^2- 4(m^2)(3k^2)}}{2m^2}= \frac{4km\pm \sqrt{4k^2m^2}}{2m^2}= \frac{4km\pm 2km}{2\m^2}$.
The two values for x are $\displaystyle x= \frac{4km+ 2km}{2m^2}= \frac{6km}{2m^2}= \frac{3k}{m}$ and $\displaystyle x= \frac{4km- 2km}{2m^2}= \frac{2km}{2m^2}= \frac{k}{m}$.
Since $\displaystyle x= \omega^2$, $\displaystyle \omega= \pm\sqrt{x}= \pm x^{1/2}$ giving
$\displaystyle \omega= \pm \sqrt{\frac{3k}{m}}$ and $\displaystyle \omega= \pm\sqrt{\frac{k}{m}}$.
Those are the four solutions to the quartic equation. As I said before, since a "frequency" is always positive, to solve this problem we drop the negative solutions.
The given equation is $\displaystyle m^2\omega^2- 4km\omega^2+ 3k^2= 0$. I would substitute x for $\displaystyle \omega^2$ (JeffM would use $\displaystyle x= m\omega^2$- either works). In terms of x, the equation becomes $\displaystyle m^2x^2- 4kmx+ 3k^2= 0$, a quadratic equation in $\displaystyle x$. By the quadratic formula, $\displaystyle x= \frac{4km\pm \sqrt{16k^2m^2- 4(m^2)(3k^2)}}{2m^2}$$\displaystyle = \frac{4km\pm \sqrt{4k^2m^2}}{2m^2}$$\displaystyle = \frac{4km\pm 2km}{2m^2}$.
The two values for x are $\displaystyle x= \frac{4km+ 2km}{2m^2}= \frac{6km}{2m^2}= \frac{3k}{m}$ and $\displaystyle x= \frac{4km- 2km}{2m^2}= \frac{2km}{2m^2}= \frac{k}{m}$.
Since $\displaystyle x= \omega^2$, $\displaystyle \omega= \pm\sqrt{x}$ giving
$\displaystyle \omega= \pm \sqrt{\frac{3k}{m}}$ and $\displaystyle \omega= \pm\sqrt{\frac{k}{m}}$.
Those are the four solutions to the quartic equation. As I said before, since a "frequency" is always positive, to solve this problem we drop the negative solutions.
I did not mean to imply in any way, shape, or form that I disagreed with Hall's method of solution. His method and mine achieve exactly the same result and in fairly similar ways. I merely found my way to be a bit more intuitive, but intuition is in the mind's eye of the beholder.
As I said, the mechanics are not hard. What may be hard is to see that a substitution may turn the quartic into a quadratic. For some reason, substitution as a technique for getting standard forms is not usually taught until calculus.