Prior to learnig "formulas" that simplify the task, you should learn the basic concepts involved. Here, you have a matrix representing a linear transformation in a given (ordered) basis. The basic point is this- using a given basis, the "representation" of the basis vector, themselves, in that basis, is <1, 0, 0>, <0, 1, 0>, and < 0, 0, 1>. It should be easy to see that multiplying a matrix by any one of those gives just a column of the matrix. For example,
the second column of the matrix.
So- to find the matrix representation of the linear transformation in this new basis, multiply the matrix by each basis vector in turn and write the result in terms of the new basis. For example,
and now, we need to write that in terms of the new basis: <-1, 2, 0>= a<-1, 1, 0>+ b<1, 1, 0>+ c<0, 0, 1>= <-a+ b, a+ b, c> so we have three equations: -a+b= -1, a+ b= 2, and c= 0. Adding the first two, 2b= 1 so b= 1/2 and we have a= 2- 1/2= 3/2. In terms of our new basis, the first column of the matrix is
You can do the same with <1, 1, 0> and <0, 0, 1> to find the second and third columns.
Of course, if you have a number of matrices to change from one basis to another, you would want a general formula to use. That's where the "change of basis matrix" comes in- we want a matrix that will change <-1, 1, 0> to <1, 0, 0>, <1, 1, 0> to <0, 1, 0>, and <0, 0, 1> to <0, 0, 1>. That is, we want
The last, of course, tells us that the last column is just as we would expect since the third vector in each basis is the same.
The first two give the equations and , and . Adding the first and third equations, so and then . Adding the the second and fourth equations, so and then . That is, the "change of basis matrix" is
I am going to call this "change of basis matrix", A.
That is exactly your "A" matrix except that mine is divided by 2. Now, suppose we have a vector, v, written in terms of the new matrix. In order to determine what the linear transfromation does to v, we could do the following: multiply by the inverse of A, to go from the new basis to the old, apply the matrix form, T, in that basis, then multiply by A to go from the old basis to the new. That would be . Of course, my factor of "1/2" in A would introduce a factor of "2" in which would cancel in this product. That is, your "A" is as good as mine!
However, you have where I have and that is an important difference. if and only if A is "orthogonal" which will be true if and only if both bases are "orthonormal"- that is they all have length 1 and have dot product 0 with each other. In this case, your first basis, <1, 0, 0>, <0, 1, 0>, and <0, 0, 1>, is orthonormal but your second, <-1, 1, 0>, <1, 1, 0>, and <0, 0, 1> is not. All vectors have 0 dot product with each other but the length of the first two is 2, not 1.