It may seem plausible to you that since $gHg^{-1}$ and $H$ share many of the same properties, they must be the same. However, I assure you this is NOT the case. Let's explore this a little bit, before we finally answer the question.

Suppose $G$ is abelian. Then $Hg = \{hg: h \in H\} = \{gh: h \in H\} = gH$, since for every $g,h \in G,\ gh = hg$.

It follows, then, that $gHg^{-1} = \{ghg^{-1}:h \in H\} = \{gg^{-1}h: h \in H\} = \{h: h\in H\} = H$. So if we want to have ANY hope of finding a counter-example, we need to look in a non-abelian group.

The smallest non-abelian group is of order 6, and can be written like so:

$G = \{e,a,a^2,b,ab,a^2b\}$

where we evaluate products using the 3 rules:

1. $a^3 = e$

2. $b^2 = e$

3. $ba = a^2b$ <---this rule means $G$ cannot be abelian.

For example: $(a^2b)(ab) = a^2(ba)b = a^2(a^2b)b = a^4b^2 = a^4e = a^4 = a(a^3) = ae = a$.

You may wonder whether or not this really is a group, because although we have an identity, I have not proven the product is associative, or that we even have inverses. That is a valid question, and the best defense I can offer (without writing for 3 or 4 pages) is that you think of $a$ as the mapping of the plane which rotates it counter-clockwise 120 degrees, and think of $b$ as reflection about the $x$-axis. With a little thought, you might be able to come up with 2x2 matrices that perform these tasks, and matrix multiplication is associative, and closure can be shown by showing these 6 matrices all preserve an equilateral triangle (we have 3 rotations, 3 reflections). All of these matrices are invertible. It's fun to do, you should try proving this.

But, to this matter at hand: we need to find a suitable subgroup that "behaves badly". Let's look at $H = \langle b\rangle$. Since $b \neq e$, and $b^2 = e$, we see that $b$ is of order 2, so $H = \langle b\rangle = \{e,b\}$.

Now let's look at $aHa^{-1}$.

Since $a^3 = e$, we have that: $a(a^2) = e$, so that evidently, $a^{-1} = a^2$. So we need to evaluate two products: $aea^2$ and $aba^2$ (since we have two elements in $H$).

Now $aea^2 = aa^2 = e$, so far, so good.

$aba^2 = a(ba)a = a(a^2b)a = (a^3)(ba) = ba = a^2b$. Huh.

So for THIS subgroup $H$ of this group $G$, we have $aHa^{-1} = \{e,a^2b\} \neq \{e,b\} = H$.

Note that $(a^2b)^2 = (a^2b)(a^2b) = (a^2)(ba)(ab) = (a^2)(a^2b)(ab) = a^4(bab) = a(bab) = a(ba)b = a(a^2b)b = a^3b^2 = ee = e$.

So $a^2b$ is ALSO an element of order 2, and this might help explain WHY this occurs: we have two "copies" of the same subgroup in $G$, and conjugating by $a$ sends one to the other (there is actually a third copy:

$a^2Ha = \{e,ab\}$ so conjugating by $a$ (if you don't know already: conjugating by $a$ is the mapping $g \mapsto aga^{-1}$, for any $g \in G$) sends these 3 subgroups in a 3-cycle:

$H \to aHa^2 \to a^2Ha \to H$).

So Herstein is correct, but fair warning: math texts CAN contain typographical errors, and "proof by authority" is bad form for a mathematician, you are encouraged to use logic instead .