Prove that O is not a normal subgroup of M
where O is the group of of orthogonal operators
1. A rigid motion is a map such that if X, Y are points of , then the distance from X to Y is equal to distance from m(X) to m(Y), i.e, |m(X) - m(Y)| = |X-Y|. The rigid motion of forms a group .
2. The orthogonal matrices from a subgroup of forms a subgroup of denoted by .
3. A subgroup of whose elements do not fix the origin is the translation. A translation by p is the map .
Now going back to your problem, suppose to the contrary that is a normal subgroup of . This implies that for all , . However, the set of motions fixing p involves the conjugate subgroup , which does not necessarily fix an origin (Remember that every orthogonal operator fixes an origin), contradicting that is a normal subgroup of . Thus is not a normal subgroup of . Q.E.D.
On the contrary, is a normal subgroup of . You can verify this by defining a map , given by and , where denotes origin-fixing rotations and r is a reflection in . Note that every element of m in can have forms either or else . You can see that the kernel of is , which implies that is a normal subgroup of .