its probable easiest to build up to this. Suppose i fix the dimensions at x and y. Lets find the number of JPEGs that can be made.
There are xy pixels, and each one can take 256^3 values. So the number is
Now lets vary one of the dimensions (x), while pretending the other(y) is fixed at some value. We can just add up our formula for all the possible values of x
Combinations if height is fixed at y :
Now lets vary the other dimension. We can just add up the previous formula for all the possible values of y.