Two elements

are equivalent modulo

iff

<-- Do you mean that A,B in G **are in H** iff ...?
<=>

there exists

, so that

If you now write the matrices C and B explicit and calculate CB, you can see that in order for this equation to hold, B and A must have identical second rows.

<-- if A,B are in H, then their second row is 0,1 anyway :O
So you can conclude that if A and B are equivalent, then they must have identical second rows at least or conversely: If the second rows of A and B are different, then A and B are not equivalent.

Since there are infinitely many matrices in G (i.e. invertible matrices with rational entries) which have pairwise different second rows, there are infinitely many elements that are all pairwise not equivalent and so the index [G:H] must be infinite.