I think you need to understand that, no, no one here is going to do your homework for you. You learn mathematics by **doing** mathematics, not by watching someone else do it for you.

Here's how I would approach this problem. The plane 2x+ 2y+ z= 6 can be written as z= 6- 2x- 2y or as the "position vector" \(\displaystyle \vec{r}(x,y)= x\vec{i}+ y\vec{j}+ z\vec{k}= x\vec{i}+ y\vec{j}+ (6- 2x- 2y)\vec{j}\).

The derivatives with respect to x and y are \(\displaystyle \vec{r}_x= \vec{i}- 2\vec{j}\) and \(\displaystyle \vec{r}_y= \vec{j}- 2\vec{k}\). Those vectors are in the plane and their cross product, \(\displaystyle 2\vec{i}+ 2\vec{j}+ \vec{k}\) is perpendicular to the plane and its length is the "differential of area" for the plane. In general, the vector \(\displaystyle A\vec{i}+ B\vec{j}+ C\vec{k}\) is the "vector differential of area" for the plane Ax+ By+ Cz= D.

Now, what about the orientation? You say "where \(\displaystyle \vec{n}\) is the **outer** normal to S". That would make sense for a closed surface but not for a plane! A plane has no "inside" or "outside". Which **side** of the plane is the normal to be on? Do you want this plane oriented by the "normal in the positive z direction" or in the "negative z direction"?

If positive z direction, take the dot product of \(\displaystyle \vec{F}= xy\vec{i}- x^2\vec{j}+ (x+ y)\vec{k}\) with \(\displaystyle 2\vec{i}+ 2\vec{j}+ \vec{k}\) and integrate with respect to x and y. If the negative z direction, take the dot product of \(\displaystyle \vec{F}\) with \(\displaystyle -2\vec{i}- 2\vec{j}- vec{k}\) and integrate (that will just change the sign on the answer).

The plane 2x+ 2y- z= 6, projected to the xy-plane, is 2x+ 2y= 6 or x+ y= 3. You can get the limits of integration from that.

Would we not be better served by going to the definitions directly? Because we have a plane we will have 2 flux integrals to compute: the top and bottem. Of course you address this in your post (actually, everything i'm about to say you've already addressed) but I think for the purposes of the student it's just easier to learn how to "plug and chug" so to speak (i know...i know this doesn't actually facilitate learning the material, but if it helps understand how to at least attempt these problems, why not?)

Let us write all the things we know,

Our function is defined as,

\(\displaystyle F = xy \hat i - x^2 \hat j + (x+z) \hat k \)

Our surface is defined as,

\(\displaystyle z = 6-2x-2y \)

And for the computation of our integral,

\(\displaystyle \hat N dS = +/- ( - \frac{ \partial S }{ \partial x } \hat i - \frac{ \partial S }{ \partial y } \hat j + k ) \)

Notice how we have a plus and a minus, this means we need to find both of these to get to the total flux! If you just want upward or downward, only compute the flux integral for either of those.

\(\displaystyle \hat N dS = +/- ( + 2 \hat i + 2 \hat j + 1k ) \)

Thus,

\(\displaystyle Total Flux = \iint_D F \cdot ( + 2 \hat i + 2 \hat j + 1k ) dA + \iint_D F \cdot ( - 2 \hat i - 2 \hat j - 1k ) dA \)

To get our bounds we let \(\displaystyle z = 0 \) which yields \(\displaystyle x + y = 3 \)

And,

\(\displaystyle 0 \le x \le 3 \) with \(\displaystyle 0 \le y \le 3 - x \)

At this point everything should be clear enough for you to go ahead with this problem.

HallsofIvy gave a much more mathetmatical explanation then myself, but this should allow you to tackle the problem in a brute force manner.