As for question 1, you probably mean that ~ is an equivalence relation on X, right?

To show that the quotient map has the property that for all open U in X, p(U) is open in X/~ (such a map is called an open map), you need to recall what it means for a set in X/~ to be open. Indeed, a set V in X/~ is open, iff p^{-1}(V) is open. So for an arbitrary open U in X, you need to show that p^{-1}(p(U)) is open in X.

But this set is {x in X | p(x) in p(U)} = {x in X | [x] in p(U)}, where [x] is the equivalence class of x.

Continuing, we have

= {x in X | there is a y in U, such that y~x}

= {x in X | there is a y in U and a g in G, such that g(x)=y}.

So all x for which a homeomorphism g maps x to U make up that set. In other words, the set is

cup g^{-1}(U),

where the union "cup" is taken over all homeomorphisms g in G.