how do you find an invertible matrix R such that R^(-1) AR is upper triangular..where assuming A is not diagonalizable.
eg
0 1 1
-1 2 1
0 0 1
I think you can use the QR factorization. Yup that will do it.
http://en.wikipedia.org/wiki/QR_decomposition