Actually , in the question doesn't define what is <A, B> but , I think that means; check that the group generated by all possible products of A and B is a subgroup of order 8????
because if we think that <A, B> is just an inner product then <A, B>= and it is just an element of GL_2(C) right? not a subgroup....?
i am confused when reading this thread. for example i get that B^2 = I, so that B is of order 2. furthermore i agree with the OP, and disagree with drexel28 that: AB = -BA, rather than AB = A^-2BA.
first, let's agree on a standard way to write the possible products:
since AB = -BA, whenever we get a B before an A, let's replace the BA with -AB. this means we can always put the A's first.
also, note that A^2 = -I, so A^3 = -A. so let's agree to write all instances of -A as A^3, and when the powers of A get greater than 4,
we can reduce them, since A^4 = I.
now, we know I, A, A^2 and A^3 are all different, so that's 4 elements. since B is of order 2, the only possible products left are:
B, AB, A^2B and A^3B. this tells us we have 8 elements in the set of all possible products of A's and B's.
to show this is a subgroup of GL2(C), we need only show closure, as this is a finite set.
but it should be clear that we can write [(A^j1)(B^k1)][(A^j2)(B^k2)], j1,j2 = 0,1,2 or 3 and k1,k2 = 0 or 1 (here i am using the convention that A^0 = B^0 = I).
as one of our 8 products above by pulling the A^j2 term in front of the B^k1 term, using BA = A^3B (repeatedly, if necessary).
for example: (A^3B)(A^2B) = (A^3)(BA)(AB) = (A^3)(A^3B)(AB) = (A^6B)AB = (A^2B)AB
= A^2(BA)B = A^2(A^3B)B = A^5B^2 = A.
alternatively, we can note that the rules A^4 = B^2 = I, BA = A^3B = A^-1B are the same generating relations as the dihedral group of order 8,
with <A> being the rotation group, and B a generating reflection.