Given ,
Total subject = 15
A - 5 subjects
B - 5 subjects
C - 3 subjects
D - 2 subjects
If you randomly select 8 subjects,what is the probability that :
1)Five of the eight subjects get A?
Figure out the number of integral solutions to the Diophantine equation $\displaystyle x_A+x_B+x_C+x_D = 8$ with $\displaystyle 0\le x_A\le 5, 0 \le x_B \le 5, 0\le x_C \le 3, 0\le x_D \le 2$. That will give you the total number of possible outcomes. Then, figure out the number of integral solutions to $\displaystyle x_B+x_C+x_D = 3, 0\le x_B, 0\le x_C, 0\le x_D\le 2$. That will give you the number of outcomes with five of the eight subjects getting A.
To account for the upper bounds, use Inclusion-Exclusion. First figure out the number of possibilities with no upper bounds on any of the $\displaystyle x$'s, then subtract the number of outcomes where a single variable violates the upper bound, then add the number of outcomes where exactly two variables violate their upper bounds, subtract where exactly three variables violate their upper bound, and finally add the number of outcomes where all four variables violate their upper bound.
Since there are eight subjects, and they will have one of four letters, we want to figure out how many subjects will wind up with a particular letter. So, consider the symbols oooooooo||| (eight o's and three |'s). By permuting these, we can figure out the total number of ways of taking eight people assigning letters to them (where the specific person chosen doesn't matter). There are $\displaystyle \binom{8+4-1}{8} = \binom{11}{8} = 165$ total outcomes (with no upper bound). Next, we want the number of outcomes with $\displaystyle 6\le x_A \le 8$. In other words, we want the number of integral solutions to $\displaystyle x_A+x_B+x_C+x_D = 2$ with no upper bounds on any of the variables (and the same for $\displaystyle 6\le x_B \le 8$). So, there are $\displaystyle \binom{2+4-1}{2} = \binom{5}{2} = 10$ outcomes where $\displaystyle x_A$ violates the upper bound and another 10 outcomes where $\displaystyle x_B$ violates its upper bound. Next, let's subtract the number of ways that $\displaystyle x_C$ violates its upper bound. We want $\displaystyle 4\le x_C \le 8$. So, we want the number of solutions to $\displaystyle x_A+x_B+x_C+x_D = 4$. There are $\displaystyle \binom{4+4-1}{4} = \binom{7}{4} = 35$ such outcomes. Then, we want $\displaystyle 3\le x_D \le 8$, which is given by $\displaystyle x_A+x_B+x_C+x_D = 5$. There are $\displaystyle \binom{5+4-1}{5} = \binom{8}{5} = 56$ such outcomes. There only outcomes that can violate the upper bounds for two variables are for $\displaystyle 4\le x_C$ and$\displaystyle 3\le x_D$. So, finally, we want the number of integral solutions to $\displaystyle x_A+x_B+x_C+x_D = 1$, which is $\displaystyle \binom{1+4-1}{1} = 4$. So, there are $\displaystyle 165 - (10+10+35+56) + 4 = 58$ total outcomes from choosing 8 subjects.
Next, you want the number of outcomes with 5 of the 8 getting an A: $\displaystyle x_B+x_C+x_D = 3$. There are $\displaystyle \binom{3+3-1}{3} = \binom{5}{3} = 10$ such outcomes. Then, we want the number of outcomes with $\displaystyle x_D$ violating its upper bound. There is exactly 1 outcome with $\displaystyle x_D=3$. So, that leaves 9 outcomes.
Now, we just divide: $\displaystyle \dfrac{9}{58}$ is the probability that five of the eight subjects get A.
Wait, I may have misread the problem. I was assuming the subjects were indistinguishable. If they are distinguishable, then you need to use a different algorithm.
Instead, the total number of outcomes is $\displaystyle \binom{15}{8}$ (that is the number of ways to choose 8 subjects from the group of 15). Then, the outcomes with 5 A's is given by assuming 5 of the eight are chosen, and choosing the last three from among the remaining 10 subjects: $\displaystyle \binom{10}{3}$.
That means the probability of selecting 5 of 8 with A is $\displaystyle \dfrac{\binom{10}{3}}{\binom{15}{8}} = \dfrac{8}{429}$
Another way to do this: There are a total of 15 people, 5 with "A"s, 10 not with "A"s. The probability the first person chosen has an "A" is 5/15. There are now 14 people, 4 with "A"s so the probability the next person chosen has an "A" is 4/14. Continuing like that, the probability the first five chosen have "A"s is (5/15)(4/14)(3/13)(2/12)(1/11). There are now NO "A"s left so the last three chosen must not have "A"s. The probability that the first 5 people chosen have "A"s and the next three do not is (5/15)(4/14)(3/13)(2/12)(1/11).
If we look for five "A"s and three non-"A"s in a different order we find that, although the fractions are different, we have exactly the same numerators and same denominators so the same total. All we need to find the total is multiply that number by the number of different orders of five "A"s and three non-"A"s. That is 8!/(5! 3!)= (8)(7)(6)/(3)(2)= 56. The probability of three "A"s out of 8 is 56(5/15)(4/14)(3/13)(2/12)(1/11).