First let's check that I am reading your problem correctly (and give the masses names).

Mass A has a mass of 5m and moves from left to right at speed u.

Mass B has a mass of m and moves from right to left at speed 2u.

Now if we define left to right as the positive direction we can re-state this as:

Mass A has a mass of 5m and moves with velocity u.

Mass B has a mass of m and moves with velocity -2u.

So

Mass A has momentum 5*1=5

Mass B has momentum 1*(-2)=-2

Therefore before the collision the total momentum is 5+(-2)=3.

After the collision mass B has changed direction so its velocity has changed from -2u to +2u. Its momentum has changed to +2.

The total momentum of the system is constant so mass A must have a post collision momentum of 1. Therefore, 5*u=1 and hence the velocity of mass A has reduced from u to u/5.