My notation:

the sample mean:

the sample variance:

the population mean:

the population variance:

i assume you are trying to do amethod of momentsestimate of the parameters.

background info: http://en.wikipedia.org/wiki/Method_...ts_(statistics)

Quick explanation: equate the sample moments with the theoretical population moments

According to Negative binomial distribution - Wikipedia, the free encyclopedia, the moments for this distribution are:

So

To obtain the method of moments estimator, replace all the moments in the above equation with their sample analogues.

So your method of moment estimate for r is

Can you use a similar approach to find the estimate for p?

NBI'd check the theoretical moments are correct if you aren't getting sensible answers. wikipedia is sometimes wrong about those