Let $\displaystyle G$ be a finite non-abelian group and $\displaystyle f \in \text{Aut}(G).$ Let $\displaystyle S=\{x \in G: \ f(x)=x^{-1} \}.$ Prove that: $\displaystyle |S|\leq \frac{3}{4}|G|.$
What about this approach: Suppose $\displaystyle |S|>\dfrac{3|G|}{4}$
Though $\displaystyle S$ is not necessarily a subgroup, it does contain the identity and inverses by the automorphism properties, i.e. if $\displaystyle e$ is the identity and $\displaystyle x\in S$, $\displaystyle f(e)=e;\, f(x^{-1})=f(x)^{-1}=(x^{-1})^{-1}=x$. Then in particular $\displaystyle f(f(x))=x$, and the set of fixed points of $\displaystyle f\circ f$ does form a subgroup. Since this subgroup contains more than half of the elements it follows that $\displaystyle f(f(x))=x$ for all elements of the group.
If we can somehow show that in fact $\displaystyle S=G$, we are done.