Here's how I would go about solving the problem: P is within the square (call is S) if and only if P' is within a new square S', where S' has sides aligned with the coordinate system. This new problem is easy to solve, because all you have to check is that x' and y' are both within the limits. So, here's the method:

1. Find a rotation matrix that takes S to an aligned square S'.

2. Using the same translation matrix, take P to P'.

3. Check if the new P' is in the new square S'.

Make sense? There might be other ways of doing it, but this is the method that comes to mind as the easiest.