I'm sorry but I really cannot make heads or tails out of what you are doing here! You have matrices A and B, with a "k" in them and then give "AB" that is clearly NOT the product of the previous A and B. Where did that "n" come from?
But what system of equations are you trying to solve? Is it the system you give:
x-2y+z=1, 2x+y+2z=12, 3x+2y-z=3 ? If so its "coefficient matrix" is
but that does not have anything to do with the "A" and "B" matrices from before!
Yes, you can do a problem like this by finding the inverse matrix. There are a number of ways of doing that but the simplest is "row reduction". Write the matrix and the identity matrix "side by side".
and use "row operations" to reduce the first matrix to the identity while applying those same row operations to the other matrix, getting the inverse matrix there at the end. There are three kinds of row operations, multiply or divide an entire row by a number, add or subtract a multiple of one row from another, and swap two rows. Work one column at a time.
Our objective is to get a 1 in the first column of the first rwo, the get 0s in the first column of the other two rows. There is already a 1 where we want it so we don't have to do anything to the first row. There is a 2 in the second row of the first column and we need to subtract 2 to get a 0 there. Since we have to use row operations, subtract 2 times the first row from the second row. There is a 3 in the third row of the first column- subtract 3 times the first row from the third row.
That completes the first column. Now, we have a 5 in the second column of the second row and we want a 1 there- divide the second row by 5. We have a -2 in the second column of the first row and we want a 0 there- add 2 times the new second row (after dividing by 5) to the first row. We have a 8 in the second column of the third row and we want a 0 there- subtract 8 times the new second row from the third row. Now we have
Finally, we want a 1 in the third column of the third row. There is a -4 there so divide the third row by -4. We want a 0 in the third column of the first row and there is a 1 there- subtract the new third row from the first row. We want a 0 in the third column of the second rwo and there is a 0 there- we don't have to do anything to the second row. That gives
If I haven't made a mistake, you should be able to multiply that matrix with the original coefficient matrix (in either order) and get the identity matrix. And, of course, the solution to system of equations is given by
Unless you have a large number of problems, all with the same coefficient matrix but different right sides, you don't really need to find the inverse matrix. It is quicker to set up the "augmented matrix", adding the right side of the system as a new column and row reduce
to make the first three columns and rows the "identity" matrix. The fourth column will be the x, y, z values.