suppose we want to find out how T transforms vectors in B'-coordinates.
well, all we know at the moment, is what T does to B-coordinates (the standard ones).
but whatever [T]_{B'} is, we know that the first column of [T]_{B'} is:
[T]_{B'}[b_{1}]_{B'}.
so let's find [T(b_{1})]_{B} = [T]_{B}[b_{1}]_{B}, and then figure out how to express this in B'-coordinates.
in B-coordinates, b_{1} is (1,1,1). and T(1,1,1) = (5,7,5). since this is 5(1,1,1) + 2(1,1,0) - 2(1,0,0), in B'-coordinates, this is [5,2,-2]_{B'}.
ok, now let's do the same with the 2nd B' basis vector (1,1,0). computing T(1,1,0) = (4,5,1) = 1(1,1,1) + 4(1,1,0) - 1(1,0,0), so this is [1,4,-1]_{B'}.
finally, we have T(0,0,1) = (1,2,4) = 4(1,1,1) - 2(1,1,0) - 1(1,0,0), so this is [4,-2,-1]_{B'}.
therefore, it would seem that [T]_{B'} =
$\displaystyle \begin{bmatrix}5&1&4\\2&4&-2\\-2&-1&-1 \end{bmatrix}$.
well, let's see if this checks out.
we should have [T]_{B'} = P^{-1}[T]_{B}P, where:
$\displaystyle P = \begin{bmatrix}1&1&1\\1&1&0\\1&0&0 \end{bmatrix}$
now [T]_{B}P =
$\displaystyle \begin{bmatrix}5&4&1\\7&5&2\\5&1&4 \end{bmatrix}$
(these numbers should look familiar, they are the images of the B'-basis under T in the standard basis).
and P^{-1}[T]_{B}P =
$\displaystyle \begin{bmatrix}0&0&1\\0&1&-1\\1&-1&0 \end{bmatrix} \begin{bmatrix}5&4&1\\7&5&2\\5&1&4 \end{bmatrix} = \begin{bmatrix}5&1&4\\2&4&-2\\-2&-1&-1 \end{bmatrix}$
i have NO idea what you are doing with the row-reduction in part (b).
Thank you so much, I have basically learned this lesson completely for your detailed worked out solution, and I was able to solve the rest of questions on the lesson, apparently I have a complete misunderstanding
again thank you sooooo much for the work put it in this solution