Mass A has a mass of 5m and moves from left to right at speed u.
Mass B has a mass of m and moves from right to left at speed 2u.
Now if we define left to right as the positive direction we can re-state this as:
Mass A has a mass of 5m and moves with velocity u.
Mass B has a mass of m and moves with velocity -2u.
Mass A has momentum 5*1=5
Mass B has momentum 1*(-2)=-2
Therefore before the collision the total momentum is 5+(-2)=3.
After the collision mass B has changed direction so its velocity has changed from -2u to +2u. Its momentum has changed to +2.
The total momentum of the system is constant so mass A must have a post collision momentum of 1. Therefore, 5*u=1 and hence the velocity of mass A has reduced from u to u/5.