I'm a little puzzled that '4' was attempted. Are you SURE this shouldn't be '5'?
Factors come in pairs (if you count perfect squares as 2 copies of the same thing). Example: When finding factors of 36, one mighy try '3'. Yes! 3*12. Is there, then, a need to check to see if '12' is a factor?