I think the matrix should be:
Isn't then an initial-state matrix needed to calculate the rest?
Anyway, if you have an initial state matrix, called and then the transition matrix above, called , the first state matrix, would be calculated as follows: . The second state matrix as: . The k-th state matrix:
I'm not quite sure about those Markov chains, but I hope that helps.