Transition matrix is a term used in methods like markov chains to transition between one stage to the next.
Transformation matrix is a term used if you want to move or rotate a set of points.
The mechanics of these two are pretty much the same.
In linear algebra, a change of basis matrix is often called a transition matrix (or matrix of transition).
Essentially, they represent the identity map relative of various bases.
As far as transformation matrix is concerned (again in linear algebra), I've seen the term
used for the matrix representation of some linear transformation, say T: V -> W, relative to specific bases in V and W.