i need mathematical proof of Probability mass function of negative binomial distribution.
all trials are independent. The chance of the kth failure happening on the nth trial is:
P(k-1 failures in n-1 trials) x P(nth trial is a failure)
The first term is just the "regular" binomial pmf (with suitable parameters) and the second term is (1-p).
You need to do the research. Go to your institute's library. Find this book: Amazon.com: Univariate Discrete Distributions (Wiley Series in Probability and Statistics) (9780471272465): Norman L. Johnson, Adrienne W. Kemp, Samuel Kotz: Books
Read it.