$$\int\int xydS = \int\int xydS_1 + \int\int xydS_2+\int\int xydS_3 + \int\int xydS_4$$
Since you are told the tetrahedron is regular, you know
$$\int\int xydS_1 = \int\int xydS_2 = \int\int xydS_3 = \int\int xydS_4$$
This means, you only need to calculate the area of one side and multiply by 4. Assume $S_1$ is bounded by these lines:
$$x=0$$
$$y=\sqrt{3}x$$
$$y=\sqrt{3}(a-x)$$
So, the area would be:
$$\int\int xy dS_1 = \int_0^{a/2}\sqrt{3}xdx + \int_{a/2}^a \sqrt{3}(a-x)dx = \dfrac{\sqrt{3}a^2}{4}$$
Multiplying by 4 to get the complete area gives $\sqrt{3}a^2$ as desired.
Thanks a lot SlipEternal. Integrating each triangle alone is a nice way, but I need to integrate the whole regular tetrahedron at once going from 0 to h (height).
We did that method for the volume of a solid regular tetrahedron and it worked fine since we had mass everywhere inside the object.
For the area, I think it is very complicated to do that since there is no mass inside the object! But I am wondering, is it possible to set an integral from 0 to h?
Why would I want to do that?
I am trying to find the mass moment of inertia of a hollow regular tetrahedron.
I can do that only if I have a slice of the height. Then integrating this slice from 0 to the complete height!