I think this is a huge assumption to make, but I think this is true. I'm not sure how to bridge the gap.
I'm still kind of confused why for some group H and some $\displaystyle h \in H, aha^{-1} = h' \in H$ with $\displaystyle h' \neq h$ isn't a possibility.