try to break down:
1) the number of ways the square's edges can be painted with k colors, for k = 1,2,3,4. how does knowing the symmetries of the square help here?
2) the number of ways we can choose k colors from 6.
your answer for (1) is incorrect.
suppose we have just one color. then all ways of painting the square's edges are indistinguishable.
suppose we have 2 colors. there are 6 different ways to choose 2 edges out of 4. this gives 6 different ways to paint the square with any 2 given colors.
another way to see this, is that we can either a) paint two adjacent sides, or b) paint two opposite sides. adjacency is preseved under rotation,
but the vertices are not. this gives us 4 ways (one for each rotation) to paint adjacent sides. rotation also preserves opposite sides, but "rotation squared"
gives us a square indistinguishable from our original square, so we only get 2 ways to paint opposite sides.
suppose we have 3 colors. one color gets painted to two sides, there are 6 different ways for this to happen, and 3 colors for it to happen with.
for each of those 18 ways, there are 2 ways to choose to paint the remaining 2 sides, giving 36 ways to paint the square with 3 colors (and we haven't even chosen the colors yet!).
finally, we have 4! ways to paint the square with 4 colors: 4 choices for the 1st color, having done that 3 choices for the 2nd color, and 2 choices for the 3rd color.