Let a,b be automorphisms of G, and let ab be the product of a and b as mappings on G. Prove that ab is an automorphism of G, and that the inverse of a is an automorphism of G, so that the set of all automorpisms of G is itself a group.
Are you sure you want your group operation in Aut(G) to be pointwise multiplication? I think you would want composition to be the group operation.
For instance, let $\displaystyle \sigma^1, \sigma^2 \in Aut(G)$. And consider the following.
$\displaystyle \sigma^1(xy)\sigma^2(xy)=\sigma^1(x)\sigma^1(y)\si gma^2(x)\sigma^2(y)$
And what you want to say is that this should be
$\displaystyle \sigma^1(x)\sigma^2(x)\sigma^1(y)\sigma^2(y)$
But this requires G to be commutative since you need to switch the two inner terms if it is to be a homomorphism.
If instead you use composition
$\displaystyle \sigma^1(\sigma^2(xy))=\sigma^1(\sigma^2(x)\sigma^ 2(y))=\sigma^1(\sigma^2(x))\sigma^1(\sigma^2(y))$ which is a homomorphism. Furthermore the composition of bijections is a bijection, so it is an isomorphism and clearly it still goes from G to G so it is an automorphism.
Inverses are particularly clear for function composition as $\displaystyle \sigma \in Aut(G)$ necessarily means it is an isomorphism from G to G and therefore has an inverse that is an isomorphism $\displaystyle \exists \sigma^{-1}\in Aut(G)$, this clearly goes from G to G in the oposite direction and gives you the identity map on G. That is $\displaystyle \sigma(\sigma^{-1}(x)=Id(x)=\sigma^{-1}(\sigma(x))$
Associativity and Identity are clear. (composition of functions is associative and the identity is clearly an automorphism from G to G.)
I hope this helps.
Let a be an automorphism of G such that $\displaystyle a:G \rightarrow G$ defined by $\displaystyle a(g) = \sigma_a \cdot g$ where $\displaystyle \sigma_a,g \in G$; let b be an automorphism of G such that $\displaystyle b:G \rightarrow G$ defined by $\displaystyle b(g) = \sigma_b \cdot g$ where $\displaystyle \sigma_b,g \in G$.
Let $\displaystyle ab:G \rightarrow G$ defined by $\displaystyle ab(g) = \sigma_{ab} \cdot g$ where $\displaystyle \sigma_{ab},g \in G$ and $\displaystyle \sigma_{ab}=\sigma_{a}\cdot \sigma_{b}$. If $\displaystyle g_1 = g_2$, then $\displaystyle ab(g_1)=\sigma_{ab}(g_1) = \sigma_{a}(\sigma_{b}(g_1))=\sigma_{a}(\sigma_{b}( g_2))=ab(g_2)$ for $\displaystyle g_1, g_2 \in G$. Thus, $\displaystyle ab$ is well-defined and it is an automorphism since $\displaystyle (ab)(ab)^{-1} = 1_{G}$ and $\displaystyle (ab)^{-1}(ab) = 1_{G}$.
Let $\displaystyle a^{-1}:G \rightarrow G$ defined by $\displaystyle a^{-1}(g) = \sigma_a^{-1} \cdot g$ where $\displaystyle \sigma_a^{-1}, g \in G$. Since a is an automorphism, $\displaystyle a^{-1} $ is clearly automorphism.
Now it remains to show that the set of all automorphisms of G itself a group. The identity element of this group is $\displaystyle 1_{G}$, which maps an element of G to itself. As shown above, the inverse of an automorphism of a is $\displaystyle a^{-1}$. It is left to you to show the associative law holds for this group.