<a> is cyclic for any element a of a group G. it just isn't necessarily the whole group G.

i presume by Z*13, you mean: (Z_{13})*, the group of units of the ring Z_{13}, where the operation is multiplication modulo 13.

for any finite group G, and any subgroup H of G, [G:H] = |G|/|H|.

the order of (Z_{13})* is indeed 12, since 13 is prime, every non-zero element of Z_{13}is a unit.

it is easy to calculate <5> directly:

5^{2}= 12 (since 25 = 12 (mod 13)).

5^{3}= (12)(5) = 8 (since 60 = 8 (mod 13)).

5^{4}= (8)(5) = 1 (since 40 = 1 (mod 13)).

thus <5> has order 4.

so [(Z_{13})*:<5>] = 12/4 = 3.

now the index of a subgroup H is the number of left (or right) cosets of H. since (Z_{13})* is abelian, it doesn't really matter which cosets we look at (they are the same). i'll look at the left cosets:

<5> = {1,5,12,8} <--i will call this coset "E", since it's the identity of the quotient group (Z_{13})*/<5>

2<5> = {2,10,11,3} <--let's call this "A"

4<5> = {4,7,9,6} <--let's call this "B"

our multiplication table will start off like this:

* | E A B

---------

E | E A B

A | A

B | B

so how do we find A*A? an easy way is to take any two elements of A = {2,10,11,3} and multiply them (mod 13). whichever coset that's in, is what A*A is.

2*2 = 4, so A*A = B. (we could pick ANY two, we'd get the same answer. let's pick 10 and 11:

(10)(11) = 121 = 117 + 4 = 9*13 + 4, so (10)(11) = 4 (mod 13). or (2)(10) = 20 = 7 (mod 13), and 7 is in B).

so to find A*B, we can just pick the coset that 8 lies in, which is E. since (Z_{13})* is abelian, B*A = E, as well.

the only product left to compute is B*B. and since 4*4 = 16 = 3 (mod 13), and this is in A, we conclude B*B = A.

so our finished multiplication table looks like this:

* | E A B

---------

E | E A B

A | A B E

B | B E A