Re: How to estimate Variance of a Poisson distribution?

Quote:

Originally Posted by

**laban1** I prefere to look at it as

__n is always the number of timeslots - one sample per timeslot__

Sometimes I dont get to sample every minute in some areas.

In fact, the timeslots are directly given by the sample-rate.

If i get 6 samples per hour my timeslots are 10min per definition. If i get 60 samples per hour my timeslots are 1min.

Ok then I suggest you use 10 min time intervals to avoid getting lots of zeros. If you have 6 timeslots per hour then you can measure for 10 hours to get 60 samples. Of course the more samples you can get the better

Quote:

That was bad news I think! __If I find the constant, then I could subtract it and then do the Pearson X^2 ?__

On a Poisson, What is over-dispersion really comming from, finding more high values weighting the tail than for the polulation? I feel it would be the opposite. or is it due to noise that squared get higher weight since that is usually not Zero centered ?

If you found the constant then you could subtract it from every measurement and make a confidence interval for the mean. Since you can change your time intervals to 1 minute it should be very easy to find the constant if you are only monitoring 10 houses.

Re: How to estimate Variance of a Poisson distribution?

Quote:

On a Poisson, What is over-dispersion really coming from, finding more high values weighting the tail than for the population? I feel it would be the opposite. or is it due to noise that squared get higher weight since that is usually not Zero centered ?

Yes the over-dispersion comes from observing the higher values more often than you would expect if the system was a poisson distribution exactly.

Re: How to estimate Variance of a Poisson distribution?

Hi again and thanks for your input!

a)Any thoughts about my reasoning about ci mean / ci variance?

b)"Yes the over-dispersion comes from observing the higher values more often than you would expect if the system was a poisson distribution exactly."

What I was hoping for was if you had some sort of physical explaination on why this "often" is the case.

c)I'm thinking of two ways on using/looking at the data.

First, is taking each night in a month and make an estimation of the mean for each night. And then making some "mean of mean"

Or

Gathering all month in one "gigantic" experiment.

Any thoughts on which one would be preferable?

d) Do you have any experience/ thoughts on using Pearson's?

3) How would I use this Pearson? If Pearson would e.g. be 1,2 then I should divide my estimation of the mean by variance by 1,2 to get a "truer" value of the mean?

4) Lets say I have 60 timeslots each night and I have 30 nights in a month. Should I do this Pearson value for each night, somewhat different every night, correct each estimate of mean by variance, with pearson, that would be 30 pearson calculations over a month period?

Regards!

Re: How to estimate Variance of a Poisson distribution?

Quote:

Originally Posted by

**laban1** Hi again and thanks for your input!

a)Any thoughts about my reasoning about ci mean / ci variance?

I'm not sure which part of your reasoning you are referring to.

Quote:

b) What I was hoping for was if you had some sort of physical explaination on why this "often" is the case.

I think it is just because unknown factors make observing those rare values in the tail of the Poisson distribution more common. It is not the first time I have heard that tail values are often more frequent than expected but I'm not sure why unknown factors tend to make tail values more likely instead of less likely. Maybe it is because when unknown factors increase the tail values a little bit it is more noticable than when unknown factors increase the common results a little bit. (example of an unknown factor for your study could be when it is half-time of a football game then many people in the 10 houses might get up and use the toilet).

Quote:

c)I'm thinking of two ways on using/looking at the data.

First, is taking each night in a month and make an estimation of the mean for each night. And then making some "mean of mean"

Or

Gathering all month in one "gigantic" experiment.

Any thoughts on which one would be preferable?

One gigantic experiment would be better than getting the average of 30 variances or means.

Quote:

d) Do you have any experience/ thoughts on using Pearson's?

3) How would I use this Pearson? If Pearson would e.g. be 1,2 then I should divide my estimation of the mean by variance by 1,2 to get a "truer" value of the mean?

once you get $\displaystyle \phi=\frac{\chi^2}{n-p}$ then you should divide your estimation of the variance by $\displaystyle \phi$ to get an estimation of the mean.

Quote:

4) Lets say I have 60 timeslots each night and I have 30 nights in a month. Should I do this Pearson value for each night, somewhat different every night, correct each estimate of mean by variance, with pearson, that would be 30 pearson calculations over a month period?

You should do one calculation of the Pearson value after 30 nights

Re: How to estimate Variance of a Poisson distribution?

Hi again!

a) Any thoughts about my reasoning about ci mean / ci variance?

I'm not sure which part of your reasoning you are referring to.

Why would that be proof saying anything about the c.i. of variance? (the ci of mean) If you think about it, mean and median are very closely related, indeed if the population is not "screwed" the result is interchangeable. But, the variance is taken from the numbers squared, so the generalisation i feel would be very bold. If the solution is that easy why are they not telling that in the article. Simply "the c.i. of variance is always the same as c.i for mean in a poisson" Could you help me find a believable article saying that?

One "proof" that saying that could be wrong

If we just for one moment compere to ci of __Norm-distr__. Then I guess you know that the__ c.i of mean is T-distr__, that is, symetrical Norm-alike and becoming more or less a Norm if the no samples is 30 or more. Now, if we look at the __c.i. of variance it is as you know X^2 __and not symetrical at all ! But can be said to be Norm-alike and becoming more or less a Norm if the no samples is 50 or more. (Compare the numbers 30 and 50, I believe this is an indication on ci of Var being greater.)

Now in a the poisson; If __λ=np is greater than about 10, then the normal distribution is a good approximation__. So for np>10 the ci for mean and variance are not the same.Futhermore let np=8 to show some true "Poissonic" behaivoir then the ci of the X^2 would not hold true, but the basic shape of the X^2 would be intact.

-----------------------------------

You should do one calculation of the Pearson value after 30 nights

One thing, when I'm setting up my model to compere

with my measured data, how do I know my model is right?

Since I'm gathering data to get the model, how do I look at this?

I just need Lambda =n*p, right but if I don't?

Thanks again, for your feedback on Pearson! :)

Re: How to estimate Variance of a Poisson distribution?

Quote:

Originally Posted by

**laban1** One "proof" that saying that could be wrong

If we just for one moment compere to ci of __Norm-distr__. Then I guess you know that the__ c.i of mean is T-distr__, that is, symetrical Norm-alike and becoming more or less a Norm if the no samples is 30 or more. Now, if we look at the __c.i. of variance it is as you know X^2 __and not symetrical at all ! But can be said to be Norm-alike and becoming more or less a Norm if the no samples is 50 or more. (Compare the numbers 30 and 50, I believe this is an indication on ci of Var being greater.)

Now in a the poisson; If __λ=np is greater than about 10, then the normal distribution is a good approximation__. So for np>10 the ci for mean and variance are not the same.Futhermore let np=8 to show some true "Poissonic" behaivoir then the ci of the X^2 would not hold true, but the basic shape of the X^2 would be intact.

I said before that the confidence interval of the mean would not be the same size as the c.i. of the variance. In my example the c.i. of the mean was smaller than the c.i. of the median. I can not find a proof about the confidence interval of the variance also being a confidence interval of the mean but the people at stack exchange agree with me that it is a confidence interval for the value $\displaystyle \lambda$ whether that is the mean or the variance Stack exchange

Re: How to estimate Variance of a Poisson distribution?

Hi!

Could you comment were my reasoning above is faulty in your oppinion?

From Sack exchange (your link futher down)

"Your approach is basically correct but heavily __depends on the strong distributional assumption __ you are making. If it is violated, even for very large samples, the confidence regions wont have the stated coverage probabilities. That's why statisticians try to avoid such reasoning if there are more robust methods available "

It's part of my reasoning:

The mean is always a mean no matter of distribution. That you estimating mean from variance means at least two drawbacks.

1. You are making a strong distributional assumption, Is your reality a true mathematical poisson?

2. Do have sufficiant and reliable data (over dispersion suggest not)

3. The squared propery of the variance makes measurements more sensitive to outliers.

For me it's quite obvious, I'd like to here why you think this is wrong.

Thanks for your interest!

Re: How to estimate Variance of a Poisson distribution?

Quote:

Originally Posted by

**laban1** The mean is always a mean no matter of distribution. That you estimating mean from variance means at least two drawbacks.

1. You are making a strong distributional assumption, Is your reality a true mathematical poisson?

2. Do have sufficiant and reliable data (over dispersion suggest not)

3. The squared propery of the variance makes measurements more sensitive to outliers.

1. Yes I did make that assumption, its up to you if you want to make it too. Sometimes you have to make assumptions in order to simplify things enough to be able to calculate it.

2. I don't know, is your data reliable? If the system is a perfect poisson distribution then it should be.

3. It is more sensitive so the confidence interval would be bigger to account for that. The chance that the variance lies outside a 95% confidence interval is still 5%.