First thing is you're doing the right thing by differentiating the distance squared, because it will have the same critical points as the distance, but just as a note the distance between two points is . You're working with . But you've set up your problem correctly and differentiated correctly, so now set the derivative equal to zero and find your critical point(s)!