suppose we want to find out how T transforms vectors in B'-coordinates.
well, all we know at the moment, is what T does to B-coordinates (the standard ones).
but whatever [T]B' is, we know that the first column of [T]B' is:
so let's find [T(b1)]B = [T]B[b1]B, and then figure out how to express this in B'-coordinates.
in B-coordinates, b1 is (1,1,1). and T(1,1,1) = (5,7,5). since this is 5(1,1,1) + 2(1,1,0) - 2(1,0,0), in B'-coordinates, this is [5,2,-2]B'.
ok, now let's do the same with the 2nd B' basis vector (1,1,0). computing T(1,1,0) = (4,5,1) = 1(1,1,1) + 4(1,1,0) - 1(1,0,0), so this is [1,4,-1]B'.
finally, we have T(0,0,1) = (1,2,4) = 4(1,1,1) - 2(1,1,0) - 1(1,0,0), so this is [4,-2,-1]B'.
therefore, it would seem that [T]B' =
well, let's see if this checks out.
we should have [T]B' = P-1[T]BP, where:
now [T]BP =
(these numbers should look familiar, they are the images of the B'-basis under T in the standard basis).
and P-1[T]BP =
i have NO idea what you are doing with the row-reduction in part (b).