If you draw the picture, you'll see it is easiest to evaluate P(X^2 + Y^2 < 1) = 1 - P(X^2 + Y^2 > 1). This is just the quarter circle with radius 1 with probability pi/24.

If you have to use an integral to evaluate this, write the inequality x^2 + y^2 < 1 as y < sqrt(1 - x^2) with 0 < x < 1. The iterated integral is

int (0,1) int (0,sqrt(1 - x^2)) 1/6 dy dx = int (0,1) sqrt(1 - x^2)/6 dx.

You can use The Integrator to find the antiderivative of sqrt(1 - x^2).