I did not solve the problem yet, but I was able to simplify what you are saying.

---

You have,

aX+bY=ab-a-b

Equivaleny,

a(X+1)+b(Y+1)=ab

Call x=X+1 and y=Y+1

Thus,

ax+by=ab

Where x and y arepositive integers

Because,

X,Y wherenon-negativeimplies X+1,Y+1 are positive.

Thus, you need to show for a,b not equal to zero,

The linear diophantine equation,

ax+by=ab

Has no positive solutions.

Begin by noting the trivial solutions:

(x,y)=(b,0) and (x,y)=(0,a)

Then use them to construct your basis of solutions.

But I did not get any further.