Hi,
My earlier post on exponential notation for conjugation is used here. I'm a firm advocate of this notation; it makes lots of proofs much easier.
Actually reading that again, I still don't get how it's a subgroup...
So I realised there's an easier way to prove that it normal, but what about proving that it is an actual subgroup of AB, ie.
Also, here's my reasoning for a normal subgroup:
Since A and B are subgroups, is also a subgroup. In particular since A is abelian and then is also abelian, hence it is also a normal subgroup of G, ie.
This means
Since then
Would that work?
Suppose we have:
$abxb^{-1}a^{-1}$ with $a \in A,b \in B, x\in A \cap B$.
Since $AB$ is a group, $AB = BA$, so we can write this as:
$b'a'xa'^{-1}b'^{-1}$.
Now $A$ is abelian, and thus $a'xa'^{-1} = x$, so $b'a'xa'^{-1}b'^{-1} = b'xb'^{-1} \in B$
Also, $A \unlhd G$, so since $b' \in B \leq G$, $b'xb'^{-1} \in A$, so $abxb^{-1}a^{-1} = b'a'xa'^{-1}b'^{-1} \in A\cap B$
Proof that $AB$ is a subgroup iff $AB = BA$:
Suppose $AB$ is a group. Then, by closure, $abab = a'b'$ for any $a \in A,b \in B$ and some $a' \in A, b' \in B$.
Thus $ba = (a^{-1}a')(b'b^{-1}) \in AB$, so $BA \subseteq AB$.
Also, since $AB$ is a group, for $ab \in AB$ we have $(ab)^{-1} \in AB$, so $ab = (a'b')^{-1}$ for some $a' \in A, b' \in B$, thus $ab = (a'b')^{-1} = b'^{-1}a'^{-1} \in BA$, so $AB \subseteq BA$.
Now suppose $AB = BA$. We need to show closure and existence of inverses (that $e \in AB$ is trivial).
So take any product $aba'b'$. Since $BA = AB$, we can write $ba' = a''b''$, so $aba'b' = (aa'')(b''b') \in AB$, which shows closure.
Now if $ab \in AB$, then $(ab)^{-1} = b^{-1}a^{-1} \in BA = AB$, which shows $AB$ contains all inverses, and is thus a group.