My problem is stated as: "Determine the groups G such that is a subgroup on ." To me, appears that if G is a group, then this should always define a subgroup.
Going through the subgroup test:
Identity: All of the resulting subgroups will have an identity, namely .
Units: All elements are invertible. For any pair , the inverse is , which also exists in the subgroup.
Closed: For any , we would have , which should also exist in the subgroup.
Is there something I'm missing? The question seems to imply that in some cases, this subset will not be a subgroup, but I don't see when that would occur.