it's called latex, there is a tutorial
here. to use it, you use the "tex" tags. for example:
"[tex ]x^2 + x + \sqrt{2}[ /tex]" produces
(don't put the spaces after the tex or before the /tex)
when row-reducing, there are 3 operations we can perform:
1) switch two rows,
2) multiply one row by a non-zero number,
3) multiply one row by a non-zero number and add/subtract it from another row.
the idea is try to try get a leading 1 in each row, each leading 1 is rightward of the rows above, and we want all 0's below each leading 1 (you can also try to have 0's above each leading 1 as well, which brings the matrix into reduced row echelon form). we may wind up with 0 rows at the bottom.
in your matrix, since we already have a leading 1 in the first row, the first step is to eliminate the rest of the first column. subtracting 4 times the 1st row from the second, and then subtracting 3 times the first row form the third, we get:
(this is the augmented matrix) can you continue?