If is the vector from to whose length is 1, then the correct expression is (b). This is because is the angle between and , and is the angle between (plotted from ) and , so and . Dot product uses the angle between the two vectors. If the length of is not 1, then one has to divide by (in addition to the division by ).