I would not call that a "variable substitution", I would say you are puttting the problem in cylindrical coordinates. That is, x and y are in polar coordinates, , and . Then z= r^2 is the integrand. But where did you get that "r" in the denominator? The differential of area in polar coordinates is , not .
Since the boundary of the region in the xy-plane is the circle , r ranges from 0 to and r from 0 to a.
The volume is given by