The total number of 5-card combinations you can take from a typical 52-card deck is:
That's combination of 52 taken 5 at a time. Since the way the cards are sequenced in the hand is not relevant, the solution is a combination, not a permutation.
And then calculate the number of ways you can have a straight flush. One possible straight flush is A, 2, 3, 4, 5. Another is 10, J, Q, K, A. There are 10 straight flushes each suit (because the "lowest" card of each straight flush can range from A to 10... that's a total of 10). All in all, there are 40 straight flushes.
Hence, the probability of obtaining a straight flush is:
which is a very small number. Even smaller is the probability of obtaining a royal flush. There is only one royal flush per suit (10, J, Q, K, A), and 4 overall.
The probability of a royal flush is: