
Markov chain help please
Let X be a Markov chain with state space (a,b,c,d) and transition probabilities given by
P= [0.3 0.1 0.0 0.6
0.0 0.3 0.5 0.2
0.0 0.5 0.5 0.0
0.0 0.0 1.0 0.0]
each time the chain is in state a, a profit of 10 is made; each visit to state b yields a 30 profit;each visit to state c yields a 5 profit; and each visit to state d cost 10.
Find the following:
a) E[f(X1)X0=b]
b) E[f(X2)X0=c]
c) let the initial probabilities be given by the vector mu=(0.4, 0.2, 0.3 ,0.1).
What is the expected profit at the first step, i.e E of mu[f(X1)]?
can anyone help with ne of the above questions please. im really stuck. thank you.