I presume you have written that as a matrix differential equation as you title this "using matrix differential equations":

Surely you havesomeinstruction in this?

Start by finding the eigenvalues and eigenvectors of the matrix. Then where A is the matrix, B is a matrix having the eigenvectors of A as its columns, and D is a diagonal matrix having the eigenvalues of A on its diagonal. That is dX/dt= AX, which, after multiplying by gives . Let and the equation becomes dY/dt= DY where D is that diagonal matrix. Because D is diagonal, that is easy to solve and, once you have Y, X= AY.

HOWEVER, in this particular problem, there is no point in doing that or even writing this in terms of a matrix. The third equation is dz/dt= 2z which is easily solvable: and z(0)= C= 1. Then dy/dt= y- 5z= y- 5 which has general solution and y(0)= C+ 5= 1 so C= -4. Finally, dx/dt= x+ 3z= x+ 3 has general solution and z(0)= C- 3= 1 so C= 4.

(Okay, not all matricesarediagonalizable, and this particular one is not.)