May I know what is wrong with my workings?
There are two ways of thinking about this. A vector integral over a surface is usually written "$\displaystyle \int\int \vec{f}\cdot\vec{n}dS$" where n is a unit normal vector to the surface and $\displaystyle dS$ is the "differential of surface area".
Here, the surface is the plane x+ y+ z= 5. The way I prefer to calculate dS is to write z= 5- x- y so that the "position vector" of any point in the plane is $\displaystyle \vec{X}(x,y)= x\vec{i}+ y\vec{j}+ z\vec{k}= x\vec{i}+ y\vec{j}+ (5- x- y)\vec{k}$ in terms of the two parameters x and y. Now, the derivative vectors $\displaystyle \vec{X}_x= \vec{i}- \vec{k}$ and $\displaystyle \vec{X}_y= \vec{j}- \vec{k}$ are in the tangent plane and their cross product, $\displaystyle \vec{X}_x\times\vec{X}_y= \vec{i}+ \vec{j}+ \vec{k}$ is normal to the surface (you could have gotten that from the fact that $\displaystyle A\vec{i}+ B\vec{j}+ C\vec{k}$ is normal to Ax+ By+ Cz= D) and has length $\displaystyle \sqrt{3}$ so that $\displaystyle \vec{n}= \frac{1}{\sqrt{3}}\vec{i}+ \frac{1}{\sqrt{3}}\vec{j}+ \frac{1}{\sqrt{3}}\vec{k}$ is the unit normal and the "differential of surface area" is the length of that vector times dxdy or $\displaystyle \sqrt{3}dxdy$.
That is, here $\displaystyle \vec{n}dS= (\frac{1}{\sqrt{3}}\vec{i}+ \frac{1}{\sqrt{3}}\vec{j}+ \frac{1}{\sqrt{3}}\vec{k})\sqrt{3}dxdy= (\vec{i}+ \vec{j}+ \vec{k})dxdy$. The two "lengths", $\displaystyle \sqrt{3}$, cancel!
That is why I prefer to write "$\displaystyle d\vec{S}$" and write the integral of a vector function over a surface as $\displaystyle \int\int \vec{f}\cdot d\vec{S}$ to begin with.
Here, $\displaystyle \nabla\times\vec{f}= \vec{i}+ \vec{j}+ \vec{k}$ and $\displaystyle d\vec{S}= (\vec{i}+ \vec{j}+ \vec{k})dxdy$ so that $\displaystyle \nabla\times\vec{f}\cdot d\vec{S}= 3dxdy$ so that the integral is just 3 times the area of the surface- which is a "square of side length 2".
Note that $\displaystyle \vec{X}_x\times\vec{X}_y= -\vec{X}_y\times\vec{X}_x$ so that you have to choose in which order you will take the cross product. I took $\displaystyle \vec{X}_x\times\vec{X}_y= \vec{i}+ \vec{j}+ \vec{k}$ rather than $\displaystyle \vec{X}_y\times\vec{X}_y= -\vec{i}- \vec{j}- \vec{k}$.
That is equivalent to choosing an orientation for the surface and for its boundary. Here we are told "oriented clockwise as seen from the origin". Given that orientation for the boundary, a person "walking around" the boundary, with left side toward the interior, would have his/her head in the positive z direction. That means that we must choose the sign so that the z-component is positive.