I would first encapsulate it:
and start off with a simple expression for just to get the algebra of analysis correct.
I'm unclear about what you mean by your notation: For the inner integral, I assume that means, integrate with respect to x and replace all occurrences of x by 'x'. Same with the y. Wouldn't that just be antiderivatives? Would need constants of integrations in that case and not sure how to handle it symbolically that way. Personally, I'd do a brute-force numerical analysis on the function:
where is your function.
You can do that pretty easily in Mathematica and even create an animation file showing how the function changes as you vary the initial points . Might be fun.