I don't understand how to simplify after multiplying the numerator and denominator by the conjugate.
That is the beauty of multiplying by the conjugate, the cross terms always cancel out.
$\displaystyle (a+b)(a-b)=a^2-ab+ab-b^2=a^2-b^2$.
In this case you have $\displaystyle (\sqrt{a}+\sqrt{b})(\sqrt{a}-\sqrt{b})=\sqrt{a}^2-\sqrt{ab}+\sqrt{ab}-\sqrt{b}^2=\sqrt{a}^2-\sqrt{b}^2=a-b$
Where $\displaystyle a=\frac{x+\Delta x}{5}$ and $\displaystyle b=\frac{x}{5}$. So $\displaystyle a-b=\frac{x+\Delta x}{5}-\frac{x}{5}=\frac{x+\Delta x-x}{5}=\frac{\Delta x}{5}$
Cancel the $\displaystyle \Delta x$ and move the 5 to the denominator, also go ahead and factor out a $\displaystyle \frac{1}{\sqrt5}$ from that fraction
$\displaystyle \frac{\frac{\triangle x}{5}}{\triangle x (\sqrt{\frac{x+\triangle x}{5}}+\sqrt{\frac{x}{5}})}=\frac{1}{5\frac{1}{\sq rt{5}}(\sqrt{x+\Delta x} +\sqrt{x})}=$$\displaystyle \frac{1}{\sqrt{5}^2\frac{1}{\sqrt{5}}(\sqrt{x+\Del ta x} +\sqrt{x})}=\frac{1}{\sqrt{5}(\sqrt{x+\Delta x} +\sqrt{x})}$
Now take the limit as $\displaystyle \Delta x \rightarrow 0$ and that term goes away and you get
$\displaystyle \frac{1}{\sqrt{5}(\sqrt{x+\Delta x} +\sqrt{x})}=\frac{1}{\sqrt{5}(2\sqrt{x})}=\frac{1} {2\sqrt{5x}}=\frac{1}{\sqrt{2^2}\sqrt{5x}}=\frac{1 }{\sqrt{20x}}$