Page 1 of 2 12 LastLast
Results 1 to 15 of 16

Math Help - Poisson - Finding and Removing outlier

  1. #1
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Poisson - Finding and Removing outlier

    Hi !

    Sometimes I have lot of "zeroes" sometimes not.
    It's for practical sampling, with a lot of figures, so I'd
    rather have an easy enough approach then a perfect but complicated solution.
    Instruments sometimes give unexpected values quite off.

    How do I go about this?
    Do I have to solve
    a) upper bound: P(x>cutoff) = 0,003 (setting same level as Std3 in a Norm)
    Is there a faster/better rule of thumb here?
    It's for a practical point of view so it doesn't have to be perfect.

    b) And how do I set lower bound? seems even more tricky!

    c) When I found an outlier in a rule I set up,
    How do I go about it? Do I remove it from my sample-set completely
    directly when I discover it, or do I keep it if it
    is within expected level of probability?


    Hope you can help!
    Thanks!
    Follow Math Help Forum on Facebook and Google+

  2. #2
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,801
    Thanks
    656

    Re: Poisson - Finding and Removing outlier

    Hey laban1.

    Can you outline your sample properties? What is your distribution (assumed)? What are you using for outlier detection? (Cooks distance, something else maybe)?
    Follow Math Help Forum on Facebook and Google+

  3. #3
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    What is your distribution (assumed)?
    - Poisson (as in title), Lambda = np is fairly stable in each of the studied Areas.
    but Lambda can very from very small to very large in different Areas.

    What are you using for outlier detection? (Cooks distance, something else maybe)?
    Cooks distance? Don't know. Suppose it could be a multiple of sdt?? Hence my question.
    Follow Math Help Forum on Facebook and Google+

  4. #4
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,801
    Thanks
    656

    Re: Poisson - Finding and Removing outlier

    There is an attribute in Poisson modelling called over-dispersion.

    I think you should check it out and use your favourite software package like R or SAS to estimate the over-dispersion co-efficient and incorporate it into your analysis.

    You could throw away the 0's if you have good enough justification to throw them out, but if that is not the case then take a look at over-dispersion and consider looking at other similar methods to account for this skewed behaviour in the Poisson.

    Overdispersion - Wikipedia, the free encyclopedia

    I can't tell you what to do with the data in terms of throwing out outliers or censoring data, but I do know that for your kind of problem, over-dispersion analyses is a good first start.
    Follow Math Help Forum on Facebook and Google+

  5. #5
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    Hi!

    Thanks for your input,
    I have basic knowledge of over-dispersion. I was more into outlier.
    Follow Math Help Forum on Facebook and Google+

  6. #6
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    Any input?
    Follow Math Help Forum on Facebook and Google+

  7. #7
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,801
    Thanks
    656

    Re: Poisson - Finding and Removing outlier

    You will have to decide whether the outliers are justified to be thrown out or whether you have to use something like over-dispersion.

    We don't know enough about the context of the data to answer that for you.
    Follow Math Help Forum on Facebook and Google+

  8. #8
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    Hi! What do you need to know?
    Follow Math Help Forum on Facebook and Google+

  9. #9
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,801
    Thanks
    656

    Re: Poisson - Finding and Removing outlier

    You should decide first of all whether the outliers should stay or be thrown out.

    To do this, you need to figure out whether the outliers are representative of the data and what you are trying to answer or if they are not.
    Follow Math Help Forum on Facebook and Google+

  10. #10
    MHF Contributor
    Joined
    Nov 2010
    Posts
    1,930
    Thanks
    782

    Re: Poisson - Finding and Removing outlier

    Quote Originally Posted by chiro View Post
    We don't know enough about the context of the data to answer that for you.
    Quote Originally Posted by laban1 View Post
    Hi! What do you need to know?
    From the post just before yours, it appears chiro is asking for the context of the data. The more information you provide about the data, the better our understanding of it will become. The question "what do you need to know?" is difficult to answer from this end. We have no access to the data. We have no feel for what the data looks like, where it came from, how it is being evaluated, the circumstances of how it is obtained, potential expected causes for outliers (and possible methods for detecting them), potential dependencies that could be examined to rule out outliers, the level of accuracy you want in your findings, etc.

    Did you read up on Cook's Distance as chiro suggested? You could also look at Identifying outliers. Check and see if any of those methods seem suitable for your model. The section below titled "Working with Outliers" might also be of interest. You asked if you should delete outlier data. That practice is frowned upon.
    Follow Math Help Forum on Facebook and Google+

  11. #11
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    Thanks for your reply!
    Sorry for the late feedback

    It seems pretty advanced, for me, Cooks distance and all !
    Just to choose method as you point out, is not even given.

    I'm playing with an option to expand my measuring time so that instead of getting Lambda = 3
    for one hour, I could sum 4 hour and get Lambda = 12 and thus I would get approx Norm
    and could apply "3std-rule".
    I have quite a lot of data, and I expect the process to be fairly stable (Lambda stable) over time.

    Would that be an option to consider?
    Pros and cons?

    Thanks!

    PS. Simplified about data:
    I'm measuring waterflow during time of least expected flow, 1hour btw 03-04 at night.
    I have different flow-meaters and I expect there are calibration-problems as well.
    Last edited by laban1; October 13th 2013 at 01:50 AM.
    Follow Math Help Forum on Facebook and Google+

  12. #12
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,801
    Thanks
    656

    Re: Poisson - Finding and Removing outlier

    Again you have to decide whether its justified to take out the outliers or not.

    If you can't justify it, then you will probably have to resort to something like over-dispersion or a more general form of model.
    Follow Math Help Forum on Facebook and Google+

  13. #13
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    Quote Originally Posted by laban1 View Post
    Thanks for your reply!
    Sorry for the late feedback

    It seems pretty advanced, for me, Cooks distance and all !
    Just to choose method as you point out, is not even given.

    I'm playing with an option to expand my measuring time so that instead of getting Lambda = 3
    for one hour, I could sum 4 hour and get Lambda = 12 and thus I would get approx Norm
    and could apply "3std-rule".
    I have quite a lot of data, and I expect the process to be fairly stable (Lambda stable) over time.

    Would that be an option to consider?
    Pros and cons?

    Thanks!

    PS. Simplified about data:
    I'm measuring waterflow during time of least expected flow, 1hour btw 03-04 at night.
    I have different flow-meaters and I expect there are calibration-problems as well.
    Any input on this?
    Follow Math Help Forum on Facebook and Google+

  14. #14
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,801
    Thanks
    656

    Re: Poisson - Finding and Removing outlier

    You may not necessarily get normality by using a higher rate.

    You should look at asymptotic results, particular with regard to the deviance statistic (which is a chi-square statistic) if you have a big enough sample.
    Follow Math Help Forum on Facebook and Google+

  15. #15
    Junior Member
    Joined
    Sep 2013
    From
    Sweden
    Posts
    27
    Thanks
    1

    Re: Poisson - Finding and Removing outlier

    Quote Originally Posted by chiro View Post
    You may not necessarily get normality by using a higher rate.
    Hi! Thanks for your input!
    Why would I not get approx Norm?
    Follow Math Help Forum on Facebook and Google+

Page 1 of 2 12 LastLast

Similar Math Help Forum Discussions

  1. Replies: 3
    Last Post: November 2nd 2012, 11:03 AM
  2. Poisson Process and finding E and Var?
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: March 21st 2011, 03:28 AM
  3. Replies: 4
    Last Post: February 9th 2011, 01:56 PM
  4. Basic Stat- outlier detection
    Posted in the Statistics Forum
    Replies: 0
    Last Post: March 18th 2010, 09:46 PM
  5. [SOLVED] Outlier Impact
    Posted in the Statistics Forum
    Replies: 7
    Last Post: June 26th 2009, 04:25 AM

Search Tags


/mathhelpforum @mathhelpforum