If $g\ne g^{-1}$ then $(g^{-1})\ne (g^{-1})^{-1}$. In other words the elements which aren't idempotent come in pairs.