Two elements
are equivalent modulo
iff
<-- Do you mean that A,B in G are in H iff ...?
<=>
there exists
, so that
If you now write the matrices C and B explicit and calculate CB, you can see that in order for this equation to hold, B and A must have identical second rows.
<-- if A,B are in H, then their second row is 0,1 anyway :O
So you can conclude that if A and B are equivalent, then they must have identical second rows at least or conversely: If the second rows of A and B are different, then A and B are not equivalent.
Since there are infinitely many matrices in G (i.e. invertible matrices with rational entries) which have pairwise different second rows, there are infinitely many elements that are all pairwise not equivalent and so the index [G:H] must be infinite.