Probability is more or less applied mathematics. At higher level, the stuff in probability is quite theoretical and doesn't look like the probability of high school. The latter can be compared to the matter of word problems. You have to understand the text, and then take the information from it. After that, there is, in my opinion, another difficulty : it's very easy to understand it wrongly, because tiny elements can change all the stuff.
I think probability in high school is separated from the mass of the mathematics because it doesn't use what you do in calculus, algebra etc... In fact, it is isolated itself from the other areas. And maybe this contributes to students who like calculus or algebra not liking probability because it's a bit apart