Retreieve X & Y Values based on which side of the square you are closer to?

So i have a problem that i dont really know how to figure out per se.

I have a Box on a X/Y plane but for the sake of the conversation im going to say i have 4 points on an X/Y plane. There will be a dot at random place somewhere in this box how do i figure out which dot this random dot is close to?

Im producing a game that if an object intersects this box its going to send it outside of the box based on which side its closer to;