Prove that the set of symmetries of a figure F in the plane forms a group
how do i define each element of the set ?
Here's one way...
Let $\displaystyle V(F)=\{v_1,\ldots, v_n\}$ be the set of vertices of $\displaystyle F$. Then you can think of the group of symmetries as bijections (permutations) of $\displaystyle V(F)$. Your law of composition is then function composition.
For example, if your figure is a square, then let $\displaystyle V(F)=\{1,2,3,4\}$, where the positions of 1,2,3,4 in the plane are (1,1), (-1,1), (-1,-1), (1,-1), respectively. What are the symmetries of the square? We have rotations and reflections:
Rotations (counter clockwise):
$\displaystyle \pi/2\leftrightarrow (1234)$,
$\displaystyle \pi\leftrightarrow (13)(24)$,
$\displaystyle 3\pi/2\leftrightarrow (1432)$.
Reflections:
through x-axis $\displaystyle \leftrightarrow (14)(23)$,
through y-axis $\displaystyle \leftrightarrow (12)(34)$,
through (2,4)-diagonal $\displaystyle \leftrightarrow (13)$,
through (1,3)-diagonal $\displaystyle \leftrightarrow (24)$,
and of course we can leave the poor square alone=identity.
The set of vertices can also be changed to the set of edges of the figure, and when in >2 dimensions, you can consider the set of faces too. There are other ways to define such a group, but I find this the most intuitive.