Figure out the number of integral solutions to the Diophantine equation with . That will give you the total number of possible outcomes. Then, figure out the number of integral solutions to . That will give you the number of outcomes with five of the eight subjects getting A.

To account for the upper bounds, use Inclusion-Exclusion. First figure out the number of possibilities with no upper bounds on any of the 's, then subtract the number of outcomes where a single variable violates the upper bound, then add the number of outcomes where exactly two variables violate their upper bounds, subtract where exactly three variables violate their upper bound, and finally add the number of outcomes where all four variables violate their upper bound.

Since there are eight subjects, and they will have one of four letters, we want to figure out how many subjects will wind up with a particular letter. So, consider the symbols oooooooo||| (eight o's and three |'s). By permuting these, we can figure out the total number of ways of taking eight people assigning letters to them (where the specific person chosen doesn't matter). There are total outcomes (with no upper bound). Next, we want the number of outcomes with . In other words, we want the number of integral solutions to with no upper bounds on any of the variables (and the same for ). So, there are outcomes where violates the upper bound and another 10 outcomes where violates its upper bound. Next, let's subtract the number of ways that violates its upper bound. We want . So, we want the number of solutions to . There are such outcomes. Then, we want , which is given by . There are such outcomes. There only outcomes that can violate the upper bounds for two variables are for and . So, finally, we want the number of integral solutions to , which is . So, there are total outcomes from choosing 8 subjects.

Next, you want the number of outcomes with 5 of the 8 getting an A: . There are such outcomes. Then, we want the number of outcomes with violating its upper bound. There is exactly 1 outcome with . So, that leaves 9 outcomes.

Now, we just divide: is the probability that five of the eight subjects get A.