Solve analytically by second order diff equations
Suppose that, at step n, the population contains r(n) "regular members" and m(n) "mutants". We are told that a mutant replaces are regular member 50% of the time, that a regular member replaces a mutant 40% of the time. That is, next year the number of mutants can be modeled as m(n+1)= m(n)+ .5r(n)- .4m(n)= .6m(n)+ .5r(n) and the number of regular members by r(n+ 1)= r(n)+ .4m(n)- .5r(n)= .5r(n)+ .4m(n).
Notice that, by adding those two equations, m(n+1)+ r(n+1)= m(n)+ r(n). That is, the total population stays constant.
We can convert those two difference equations into a single difference equation by taking another step. r(n+2)= .5r(n+1)+ .4m(n+1)= .5r(n+1)+ .4(.6m(n)+ .5r(n))= .5r(n+1)+ .24m(n)+ .2r(n). Now, from the equation r(n+1)= .5r(n)+ .4m(n), .4m(n)= r(n+ 1)- .5r(n) so .24m(n)= .6(.4m(n))= .6(r(n+1)- .5r(n)= .6r(n+1)- .3 r(n). Replacing .24m(n) by that, we have r(n+2)= .5r(n+1)+ .6r(n+1)- .3r(n)- .2r(n)= 1.1r(n+1)- .5(n).
The second order difference equation for r(n), the number of regular members, is r(n+ 2)= 1.1 r(n+ 1)- .5 r(n).
Solve that difference equation for r(n). Does r(n) equal 0 for any n?