(a) use the one-step subgroup test.
(b) suppose xHx^-1 = yHy^-1. prove that x and y lie in the same coset of N(H).
This is what I have so far. Look okay?
(a): For any h, h' ∈ H, (x^(-1)hx) (x^(-1)h'x) = x^(-1) (h h') x and (x^(-1)hx)^(-1) = x^(-1) h^(-1) x, so x^(-1) H x is closed under multiplication and taking inverses. Also, if e is the identity of G, x^(-1) e x = e, so e ∈ x^(-1) H x. Therefore, x^(-1) H x is a subgroup of G.
(b): I think the statement that we're being asked to prove is that the number of distinct subgroups of the form x^(-1) H x equals [G:N(H)], where N(H) is the normalizer of H in G. If S is the set of subgroups { x^(-1) H x | x ∈ G}, observe that G acts on S by conjugation:
(g, x^(-1) H x) -> g^(-1) x^(-1) H x g = (xg)^(-1) H xg.
This action is clearly transitive on S, so by the orbit-stabilizer theorem, the size of S will equal the index in G of the stabilizer of H for this action. But this stabilizer is just { x ∈ G | x^(-1) H x = H }, which is N(H), so we have the desired result.
looks good. the orbit-stabilizer appeal is a nice touch.
for (a) you don't need to show that e is in x^-1Hx. x^-1Hx is clearly non-empty since any x^-1hx is "some" element of G, for any h in H, and closure and inverses together imply that:
(x^-1hx)(x^-1hx)^-1 = e is in xHx^-1.