It appears that FM is the free vector space (equivalently, the free F-module) generated by M (we just declare each element of M to be a basis, and take all formal F-linear combinations).

Since G acts on M, each element g of G induces a permutation of the basis vectors, which is necessarily an invertible linear transformation (a change-of-basis). I'm not aware of a name for this construction, however if M = G, we obtain (via the action of left-multiplication) the left regular representation for G.

Basically, what you are doing is "extending" an action of G on M to a representation of G over FM. For example, suppose G = (Z3,+), a cyclic group of order 3, and M = {a,b,c}, and F is the field of rational numbers. Then:

FM = {pa + qb + rc: p,q,r in Q}, which is isomorphic to Q^{3}.

One possible action of Z3 on M is:

0.b = a, 0.b = b, 0.c = c

1.a = b, 1.b = c, 1.c = a

2.a = c, 2.b = a, 2.c = b

(we let Z3 act on M by permuting the elements cyclically).

the representation over FM in the basis {a,b,c} is then given by the linear transformations:

0 --> I (the identity 3x3 matrix in the basis {a,b,c})

1 -->

[0 1 0]

[0 0 1]

[1 0 0], which sends pa + qb + rc --> qa + rb + pc

2-->

[0 0 1]

[1 0 0]

[0 1 0], which sends pa + qb + rc --> ra + pb + qc

In actual practice, since only the cardinality of M really matters, it is customary to work in the vector space F^{|M|}, with the standard basis vectors, and consider the matrices of the representation as a subgroup of the embedding of the symmetric group into GL(F^{|M|}) as standard permutation matrices.