it's often customary to write g.s instead of gs, just so we don't get confused and think gs is a product in G.

here's the thing: we want to make sure that f(gG_{s}) only depends on the coset gG_{s}, and not on which "representative" g we pick.

(in other words, we might have gG_{s}= g'G_{s}, for different g,g' and we need to be sure that we get the same value for f, that gs = g's).

now gG_{s}= g'G_{s}DOES NOT MEAN g = g'. what it DOES mean is that: g'^{-1}g is in G_{s}.

and (by the definition of G_{s}) that means:

g'^{-1}gs = s.

now what we CAN do, is have g' act on both sides (both sides are elements of S). this gives:

g'(g'^{-1}gs) = g's (*)

now, for ANY action of a group G on a set S, for g,h in G, we have:

g(hs) = (gh)s. in our situation that means (looking at the LHS of (*)):

g'(g'^{-1}gs) = (g'(g'^{-1}g))s = ((g'g'^{-1})g)s = (eg)s = gs, so NOW we have gs = g's, as desired.

***********

for your second problem, we need to pick two elements f(gG_{s}) in the orbit of s (Os), and show that if they are equal, they are images of the same coset of G_{s}.

in short, we need to show gs = g's implies gG_{s}= g'G_{s}(as luck/fate would have it, this is the REVERSE implication of showing f is well-defined).

now you can't "divide by s" (S is just a set, it may not have any "multiplication/division" structure).

what you CAN do, is act on both sides by g'^{-1}.

this gives:

g'^{-1}(gs) = g'^{-1}(g's), and BECAUSE WE HAVE AN ACTION:

(g'^{-1}g)s = (g'^{-1}g')s = es = s (because with a group action es = s, for all s in S).

but this tells us that g'^{-1}g stabilizes s (or fixes s, when we act on s by g'^{-1}g, nothing happens, we "stay" at s).

and THIS means that therefore g'^{-1}g is in the stabilizer of s, which is what G_{s}is.

but if g'^{-1}g is in G_{s}, then the two cosets g'G_{s}and gG_{s}are equal, and we're done.

finally, what we need to show is that the function f is ONTO Os.

that is, given an element in Os, say, t, for example, we need to find "some coset" of G_{s}, gG_{s}with f(gG_{s}) = t.

since t is in the orbit of s, t = gs, for some g in G. so why not try the coset gG_{s}(for that same g).

well, then we have f(gG_{s}) = gs (by the definition of f), and gs = t, so, yep, that works.

**************

i suspect that part of your trouble is you are having difficulty seeing how this all works. so let's find a simple G and a simple set S to actually work with.

let G be this subgroup of S_{4}: G = {e, (1 2 3 4), (1 3)(2 4), (1 4 3 2)}. this is a cyclic group of order 4.

what set shall we have G act on? well an obvious choice is S = {1,2,3,4}.

so for g = (1 2 3 4), and s = 3, for example, gs = 4. note that the only element of G that stabilizes anything is e, so for all s, G_{s}= {e}.

what is the orbit of 1? well 2 = (1 2 3 4)1, 3 = (1 3)(2 4)1, and 4 = (1 2 3 4)1, so the orbit of 1 is all of S.

so what we did above shows we have a bijection between G/{e} (= G) and O1 = S:

e ---> 1

(1 2 3 4) ---> 2

(1 3)(2 4) ---> 3

(1 4 3 2) ---> 4

let's pick a different G, and a different S. this time we'll pick S = G, and we'll let G = {e,a,a^{2},b,ab,a^{2}b}, where a^{3}= b^{2}= e, and ba = a^{2}b. this time, our action will be:

g.x = gxg^{-1}.

we can ask, what is the orbit of a? let's find out:

eae^{-1}= a

aaa^{-1}= a

a^{2}aa^{-2}= a

bab^{-1}= bab = (a^{2}b)b = a^{2}

(ab)a(ab)^{-1}= (ab)a(ab) = a(ba)ab = a(a^{2}b)(ab) = a^{3}(bab) = bab = a^{2}

(a^{2}b)a(a^{2}b)^{-1}= (a^{2}b)a(a^{2}b) = a^{2}ba^{3}b = a^{2}b^{2}= a^{2}.

so the orbit of a is {a,a^{2}}.

looking above, we see that the elements of G that stabilize a, are {e,a,a^{2}}, which is <a>. and what we did above says we have a bijection between:

G/G_{a}= G/<a> and Oa.

it is easy to see that the (left) cosets of <a> are <a> and b<a>. the bijection we established above is:

<a> <---> {a}

b<a> <---> {a^{2}}