May I know what is wrong with my workings?
There are two ways of thinking about this. A vector integral over a surface is usually written " " where n is a unit normal vector to the surface and is the "differential of surface area".
Here, the surface is the plane x+ y+ z= 5. The way I prefer to calculate dS is to write z= 5- x- y so that the "position vector" of any point in the plane is in terms of the two parameters x and y. Now, the derivative vectors and are in the tangent plane and their cross product, is normal to the surface (you could have gotten that from the fact that is normal to Ax+ By+ Cz= D) and has length so that is the unit normal and the "differential of surface area" is the length of that vector times dxdy or .
That is, here . The two "lengths", , cancel!
That is why I prefer to write " " and write the integral of a vector function over a surface as to begin with.
Here, and so that so that the integral is just 3 times the area of the surface- which is a "square of side length 2".
Note that so that you have to choose in which order you will take the cross product. I took rather than .
That is equivalent to choosing an orientation for the surface and for its boundary. Here we are told "oriented clockwise as seen from the origin". Given that orientation for the boundary, a person "walking around" the boundary, with left side toward the interior, would have his/her head in the positive z direction. That means that we must choose the sign so that the z-component is positive.