You can find the distance from the point to each of the sides, take the minimum, and go from there. Here's a link to a page that shows you how to compute the distance from a point to a line.
So i have a problem that i dont really know how to figure out per se.
I have a Box on a X/Y plane but for the sake of the conversation im going to say i have 4 points on an X/Y plane. There will be a dot at random place somewhere in this box how do i figure out which dot this random dot is close to?
Im producing a game that if an object intersects this box its going to send it outside of the box based on which side its closer to;
You can find the distance from the point to each of the sides, take the minimum, and go from there. Here's a link to a page that shows you how to compute the distance from a point to a line.