The key is that the total amount of liquid is still going to be equal to 20 liters in the end, so the final 40% solution will contain 8 liters of antifreeze.
Let's say that x is the amount of 100% solution you replace the 20% solution with.
So you want .2(20 - x) + x = 8. Just solve for x. Edit: this is the same as you came up with, and the answer is indeed 5. Perhaps your calculations went awry?