I imagine part of your confusion stems from the fact that what they did is wrong. The correct simplification is

$\cos\left(\dfrac \pi {10}\right)=\sqrt{\dfrac{20+\sqrt{400 - 4(16)(5)}}{32}}=$

$\sqrt{\dfrac{20+4\sqrt{25- 4(5)}}{32}}=$

$\sqrt{\dfrac{5+\sqrt{25- 4(5)}}{8}}=\sqrt{\dfrac{5+\sqrt{5}}{8}}$

They did come up with the right answer though. I wouldn't get to bothered by how they did it as long as you can work the algebra yourself and get the same answer.