Second: do you know what is? I am not sure either. So, can you give any real numbers as inputs to f? Lastly, there are several variants to define mod when one of the arguments is negative. E.g., one option is 5 mod -3 = 2 and another is 5 mod -3 = -1. Both are reasonable because 5 - 2 and 5 - (-1) are divisible by -3; both versions have advantages and disadvantages. Check if the mod function from your textbook or lecture notes accepts negative numbers.
To evaluate g(g(g(8))), find g(8) and call it, say, u. Next find g(u) and call it v. Finally, find g(v).