First of all, is there a typo in your transformation? It says . Like you said, if we find the nullity, we can use the dimension theorem to find the rank of T and we are finished the question.
Recall that the null space of T (from X -> Y) is the set of x element of X such that T(x) = 0. It is obvious that the zero element of Y in this case is the zero 2x2 matrix. So essentially, we need to find the solutions to two equations: and . Unfortunately I can't do this for you right now since I think there was a typo. But when you solve for this, it will yield a set of solutions. The dimension of this set is the nullity. Then apply the dimension theorem and you are done.