There are alternative ways to examine this.

Most simply is to take all the reds as "Group R", the whites as "Group W", the blues as "Group B".

Then the probability is the number of arrangements of the groups divided by the number of arrangements of the balls if you numbered them with labels from 1 to 10. That way you can see the different arrangements irrespective of their colours.

If we numbered the balls we can calculate the number of arrangements

with the balls together side by side in their own groups as

We multiply by 3! to arrange the groups around each other.

In any arrangement, the 5 reds can be arranged in 5! ways.

We cannot distinguish them if the balls are not numbered or differ in some way. So we divide the total number of arrangements by 5! to find the number of arrangements we can't distinguish due to the reds looking the same. We do the same for the whites and the same for the blues.

Hence we must divide by 5!, 3! and 2!

Therefore, the number of "indistinguishable" arrangements in total is

Hence the probability that the balls line up in their respective groups is