Assuming your arithmetic progression is $\displaystyle a_n = a+dn$ since you arbitrarily wrote a d for some reason, we can set these two expressions equal to each other:
$\displaystyle a_n = a+dn$
$\displaystyle = a+\dfrac{n}{5} = \dfrac{(m^2-a)n^2+m+n}{m^2+2m+1}$
If you are trying to solve for $\displaystyle m$, it is the solution to this quadratic equation in $\displaystyle m$:
$\displaystyle \left(a+\dfrac{n}{5}-n^2\right)m^2 + \left(2a + \dfrac{2n}{5}-1\right)m + \left(a+an^2-\dfrac{4n}{5}\right) = 0$
You can use the quadratic formula. You get a mess, but I suppose that could be the solution for $\displaystyle m$...