For the first one you can set up three equations in three unkowns. Let L = magnitude of the vector in the direction, and let the second vector be . The dot product of the two vectors is zero:
5x + y = 0
And the total x-direction displacement is 2: , and y-displacement: .
Solve for x and y and from that determine the turning point. If you draw a sketch you'll see that the point is greater than x=5.
For the second problem: the projection of AP onto AB is |AP|cos(theta). This is equal to the dot product , where is the unit vector in the AB direction. Using Pythagoras the length of the orthogonal is then: . Is this the equation you tried to use?