Let a,b be automorphisms of G, and let ab be the product of a and b as mappings on G. Prove that ab is an automorphism of G, and that the inverse of a is an automorphism of G, so that the set of all automorpisms of G is itself a group.
Are you sure you want your group operation in Aut(G) to be pointwise multiplication? I think you would want composition to be the group operation.
For instance, let . And consider the following.
And what you want to say is that this should be
But this requires G to be commutative since you need to switch the two inner terms if it is to be a homomorphism.
If instead you use composition
which is a homomorphism. Furthermore the composition of bijections is a bijection, so it is an isomorphism and clearly it still goes from G to G so it is an automorphism.
Inverses are particularly clear for function composition as necessarily means it is an isomorphism from G to G and therefore has an inverse that is an isomorphism , this clearly goes from G to G in the oposite direction and gives you the identity map on G. That is
Associativity and Identity are clear. (composition of functions is associative and the identity is clearly an automorphism from G to G.)
I hope this helps.
Let a be an automorphism of G such that defined by where ; let b be an automorphism of G such that defined by where .
Let defined by where and . If , then for . Thus, is well-defined and it is an automorphism since and .
Let defined by where . Since a is an automorphism, is clearly automorphism.
Now it remains to show that the set of all automorphisms of G itself a group. The identity element of this group is , which maps an element of G to itself. As shown above, the inverse of an automorphism of a is . It is left to you to show the associative law holds for this group.