Hi, I have tried some of key steps of your problem. You need to fill the remaining steps. Assume the same color can be used on any number of edges.

Burnside's lemma

.

The number of distinguishable regular octagons correspond to the number of orbits under G in a G-set X.

To apply symmetry to a regular octagon, we use .

Now we need to compute for each sixteen elements g in .

. (Every regular octagon is left fixed by the identity element)

. (To be invariant under r, r^7, all edges should be the same color. There are two colors, so two choices)

. (To be invariant under r^4, all edges should be same colors (oooooooo, xxxxxxxx), or different adjacent colors (oxoxoxox, xoxoxoxo), and (ooxxooxx, xxooxxoo). Here , (oxoxoxox) denotes edge 1 (between vertex 1 and 2) is colored o, edge 2 is colored x (between vertex 2 and 3), and so on.

. Diagonal flip : (Possible choices: oooooooo, xxxxxxxx, oxoxoxox, xoxoxoxo}

.....

After you find all |X_g| values, you need to sum up those |X_g| values

Finally, the number of orbits N (distinguishable regular octagons) can be acquired.

.