A is invertible real function,x is a vector
on what law
$\displaystyle (AA^tx,x)=(A^tx,A^tx)$
?
I presume you mean that A is a linear transformation- whether it is invertible is not really relevant. This is just the definition of $\displaystyle A^T$: If A is a linear transformation from vector space U to vector space V, then $\displaystyle A^T$ is defined as the linear transformation from V back to U such that $\displaystyle (Au, v)= (u, A^Tv)$ for any u in U and any v in V. The first inner product is in V and the second in U.
Starting from $\displaystyle (AA^Tx, x)$ with $\displaystyle u= A^Tx$ and $\displaystyle v= x$ that gives
$\displaystyle (AA^Tx, x)= (Au, v)= (u, A^Tv)= (A^Tx, A^Tx)$.