it's called latex, there is a tutorial

here. to use it, you use the "tex" tags. for example:

"[tex ]x^2 + x + \sqrt{2}[ /tex]" produces $\displaystyle x^2 + x + \sqrt{2}$

(don't put the spaces after the tex or before the /tex)

when row-reducing, there are 3 operations we can perform:

1) switch two rows,

2) multiply one row by a non-zero number,

3) multiply one row by a non-zero number and add/subtract it from another row.

the idea is try to try get a leading 1 in each row, each leading 1 is rightward of the rows above, and we want all 0's below each leading 1 (you can also try to have 0's above each leading 1 as well, which brings the matrix into reduced row echelon form). we may wind up with 0 rows at the bottom.

in your matrix, since we already have a leading 1 in the first row, the first step is to eliminate the rest of the first column. subtracting 4 times the 1st row from the second, and then subtracting 3 times the first row form the third, we get:

$\displaystyle \begin{bmatrix}1&-1&1&3\\0&1&-5&-6\\0&4&-1&14 \end{bmatrix}$

(this is the augmented matrix) can you continue?