I would first encapsulate it:

and start off with a simple expression for just to get the algebra of analysis correct.

I'm unclear about what you mean by your notation: For the inner integral, I assume that means, integrate with respect to x and replace all occurrences of x by 'x'. Same with the y. Wouldn't that just be antiderivatives? Would need constants of integrations in that case and not sure how to handle it symbolically that way. Personally, I'd do a brute-force numerical analysis on the function:

where is your function.

You can do that pretty easily in Mathematica and even create an animation file showing how the function changes as you vary the initial points . Might be fun.