Euclidean geometry...subgroups

By "Euclidean transformation", do you mean a map $T\colon\mathbb{R}^2 \rightarrow \mathbb{R}^2$ , such that $T(x) = Ax + b$ , where $A$ is an orthogonal matrix, and $b$ is some vector?

If so, to show that $E^*(2)$ is a subgroup of the group of Euclidean transformations, you need to show that $E^*(2)$ is closed under composition of maps, under inverses and that it contains the neutral element.

Closed under composition:

Take $t_1,t_2\in E^*(2)$ , and say $t_1(x) = O_1x+b_1$ and $t_2(x) = O_2x+b_2$ , where $O_1$ and $O_2$ are orthogonal matrices of determinant 1. Then:

$(t_1\circ t_2)(x) = t_1(t_2(x)) = t_1(O_2x+b_2) = O_1(O_2x+b_2) + b_1 = (O_1O_2)x + (O_1b_2+b_1),$

and provided that $O_1O_2$ is an orthogonal matrix of determinant 1, this composition is indeed in $E^*(2)$ . But this is true, as you may try to show.

Closed under inverses.

Let $t\in E^*(2)$ . We need to prove that the inverse $t^{-1}$ is also an element of $E^*(2).$ Suppose $t$ is of the form $t(x)=Ox+b$ , where $O$ is orthogonal of determinant 1. Then the inverse of $t$ is $t^{-1}(x) = O^{-1}x - O^{-1}b$ . Since $O^{-1}$ is again an orthogonal matrix of determinant 1 (which you may check), the inverse is an element of $E^*(2)$ as needed.

The neutral element is there.

The neutral element is the map $T$ given by $T(x)=Ix = x$ , where $I$ is the identity matrix. This is certainly an element of $S^*(2)$ .

As for the question "Determine which of the isometries are in $E^*(2)$ ", it depends on your definition of isometry. Are you isometries linear or are they just supposed to preserve the distance in $\mathbb{R}^2$ ?

All of the Euclidean transformations preserve the distance in $\mathbb{R}^2$ , and so does $E^*(2)$ in particular. If you are thinking in geometric terms, then $E^*(2)$ consists of the maps, which are rotations followed by a translation.