I = double integral over area S (xz^2 dydz + (x^2y − z^3) dzdx + (2xy + y^2z) dxdy) . z = 0 and z=(a^2-x^2-y^2)^1/2. So its a hemisphere with radius a.
I have tried it in carthesian coordinates, polar coordinates, spherical coordinates ... this is one impossible integral ...
My work so far ( its too much to write here , im so tired , studied 20 hours now i think or 21 ... )
I found the right answer with gauss's divergence law. I'ts 2/5 * pi * a^5.