The simplest way to find the matrix representation of a linear transformation in a give basis is to apply the transformation to each basis vector in turn, writing the result in terms of the basis. The coefficients give the columns of the matrix.
For example, T([1 0])= [6 4]= a[1 0]+ b[1 1] so we must have a+ b= 6 and b= 4. That is, a= 4, b= 4. The first column of T is [4 4].
T([1 1])= [1 5]= a[1 0]+ b[1 1]. Now we must have a+ b= 1 and b= 5 so a= 1- 5= -4. The second column of T is [-4 5].
The matrix representation of T in this basis is