A special case of this is the convention that 1 is not prime. If it were, then it would not be true that every positive integer can be uniquely factored as a product of primes, since .
It's not that much more complicated to say that "a positive integer is prime if its only divisors are itself and 1". But if we wanted to extend our definition to all integers, it becomes "p is prime if its only divisors are p, -p, 1, and -1". It doesn't get worse than that for integers, because 1 and -1 are the only units. But if we have more units, then we get a whole lot more divisors for any element, just by multiplying it by some combination of units.
The fact that these divisors exist for any element means they're not interesting, and we're motivated to throw them out. But it's not feasible anymore just to list them as exceptions, like we did above--there are too many of them. We could add something to our definition of irreducible to say that divisors which are units or unit multiples of the element don't count, and then weaken our definition of "unique factorization" to say that the units don't count there, either.
Or, we can just say that units aren't irreducible, and not have to change anything else. That way seems much simpler, so that's what we do.