When you see u: R x R ----> R it means that u is a function taking a point in R x R (which is R^2 or a 2D (x,y) point) and then producing a real number in R.
An example is u(x,y) = x + y. We take two real numbers x and y and produce a real number u(x,y).
Whenever you see A x B, it means that you form all pairs of points in a with points in b and your new point will be (a,b). You continue to do this if you have more cross products like say A x B x C or A x B x C x D and so on.
A rectangular domain is just some rectangular region. If the domain is 2D (in your case it is), then its just some rectangle that has its boundary aligned with the x and y axes.