could you explain which is correct way to transform first expression into vectors (a or b) and why?
If is the vector from to whose length is 1, then the correct expression is (b). This is because is the angle between and , and is the angle between (plotted from ) and , so and . Dot product uses the angle between the two vectors. If the length of is not 1, then one has to divide by (in addition to the division by ).