My guess is that it would be difficult to keep tension even across a square drum. It's the same principle as a table with three legs, versus four, five, six, etc. The more legs the more stable and even. A drum with 4 sides would have more uneven tension than one with 5, 6, 7, 10, 100... eventually as a shape has more sides (approaching infinity) it just becomes a circle, right?
Also, on the inside you would have two opposing faces that could reflect waves back against each other, causing distorted sound.
Just a guess, but that's my theory.