1. Matrix trace relations

The trace of a matrix is the sum of the entries on the leading diagonal. So for
a 2*2 matrix a11 a12
a21 a22
the trace is a11+a22.

Let a and b be matrices in SL(2,C) i.e. 2*2 matrices with entries that are
complex numbers and with determinant 1. Let us call the inverse matrices A,B

Preliminaries: show that trace is invariant under conjugation and inversion i.e. that trace(B*a*b) = trace(a)=trace(A)

Show that trace(a*b) + trace(a*B) = trace(a)*trace(b)

Show that if a*a = -I if and only if a has trace 0 (where I is the identity matrix).
Show that a^3 = I if and only if a has trace -1 and a^6=I if and only if
a has trace 1.

Let x = trace(a),y = trace(b),z = trace(a*b),w = trace(a*B)
Show that trace(a*b*A*B) = x*x+y*y+z*z-x*y*z-2
= x*x+y*y+w*w-x*y*w-2
= x*x+y*y-z*w-2
(The above relations, especially the first, are very important for geometry and in the theory of Kleinian Groups)

Now the hard part:

Suppose x^2 <> 4. Show that for n>=1 (trace(a^n*b*A^N*B)-2)/(trace^2(a^n)-4) is constant.

I got this result from the book "Outer Circles" which says that you can prove it for n=2 and then use induction. But I could not do it by induction.
Instead I found a general expression for calculating trace(a^n*w) given knowledge of trace(a), trace(w) and trace(a*w) by using a difference equation, and then proved it by brute force.

2. The book suggests that for $\frac{\mathop{tr}(A^nBA^{-n}B^{-1})-2}{(\mathop{tr}A^n)^2-4}$ we should write $A$ in standard form. For a $2\times2$ matrix $A$ of determinant $1$ with $(\mathop{tr}A)^2\neq4$ this means a matrix of the form $\begin{pmatrix}t&0\\0&t^{-1}\end{pmatrix}$. Since $\mathop{tr} CD=\mathop{tr}(U^{-1}CUU^{-1}DU)$ for any invertible matrix $U$, we may assume that $A$ is standard and $B$ is any $\textrm{SL}(2,\mathbb C)$ matrix.

Thus $A^nBA^{-n}B^{-1}=\begin{pmatrix}t^n&0\\0&t^{-n}\end{pmatrix}\begin{pmatrix}a&b\\c&d\end{pmatrix }\begin{pmatrix}t^{-n}&0\\0&t^n\end{pmatrix}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}$ $\ =\begin{pmatrix}ad-t^{2n}bc&\dots\\\dots&ad-t^{-2n}bc\end{pmatrix}$.

Thus $\mathop{tr}(A^nBA^{-n}B^{-1})-2=2ad-bc(t^{2n}+t^{-2n})-2=-bc(t^{2n}+t^{-2n}-2)$ using $ad-bc=1$.

Also $(\mathop{tr}A^n)^2-4=(t^n+t^{-n})^2-4=t^{2n}+t^{-2n}-2$. Therefore $\frac{\mathop{tr}(A^nBA^{-n}B^{-1})-2}{(\mathop{tr}A^n)^2-4}=-bc$, a constant. (Of course this assumes $t^{2n}\neq 1$.)

3. That's much shorter than my proof and will teach me not to assume avoiding multiplying matrices and using trace identities instead is always is a good plan. I notice you haven't proved it by induction either. Your remark shows the result is not actually true if A has finite order.