Results 1 to 6 of 6

Math Help - Claiming statistical significance - P-value or confidence interval?

  1. #1
    Newbie
    Joined
    Oct 2008
    Posts
    4

    Claiming statistical significance - P-value or confidence interval?

    Greetings all,

    I am testing a modification to an algorithm to see if significant
    performance gains can be observed when compared with the
    original algorithm.

    I have performed 30 trials of each on a benchmark problem and I
    have two sets of performance results to compare (higher values
    being better).

    Using a 95% confidence interval, I can observe (graphically) that
    there is no overlap between the error-bars, the modified method
    shows mean performance window "above" the original method.

    Using a paired t-test, I obtain a P-value less than 0.05.

    My question is -- are these two methods functionally equivalent?
    In a paper I'm writing I am using the confidence intervals to claim
    a statistically significant performance improvement.

    Is this sufficient or am I potentially leaving myself open to criticism
    by more experienced statisticians than myself?

    Any comments would be most welcome. Many thanks,

    dan
    Follow Math Help Forum on Facebook and Google+

  2. #2
    Flow Master
    mr fantastic's Avatar
    Joined
    Dec 2007
    From
    Zeitgeist
    Posts
    16,948
    Thanks
    5
    Quote Originally Posted by danc_ie View Post
    Greetings all,

    I am testing a modification to an algorithm to see if significant
    performance gains can be observed when compared with the
    original algorithm.

    I have performed 30 trials of each on a benchmark problem and I
    have two sets of performance results to compare (higher values
    being better).

    Using a 95% confidence interval, I can observe (graphically) that
    there is no overlap between the error-bars, the modified method
    shows mean performance window "above" the original method.

    Using a paired t-test, I obtain a P-value less than 0.05.

    My question is -- are these two methods functionally equivalent?
    In a paper I'm writing I am using the confidence intervals to claim
    a statistically significant performance improvement.

    Is this sufficient or am I potentially leaving myself open to criticism
    by more experienced statisticians than myself?

    Any comments would be most welcome. Many thanks,

    dan
    Finding a confidence interval or a P-value are equivalent ways of doing statistical inference at a given significance level. Be sure to quote the P-value you got, don't just say it's less than 0.05.
    Follow Math Help Forum on Facebook and Google+

  3. #3
    Newbie
    Joined
    Oct 2008
    Posts
    4

    Any references?

    Quote Originally Posted by mr fantastic View Post
    Finding a confidence interval or a P-value are equivalent ways of doing statistical inference at a given significance level.
    Cool, thanks, that is just what I was looking for.
    I'd like to make a citation to support this - would
    you happen to know of a specific text I could use as
    a reference?

    Cheers,
    dan
    Follow Math Help Forum on Facebook and Google+

  4. #4
    Flow Master
    mr fantastic's Avatar
    Joined
    Dec 2007
    From
    Zeitgeist
    Posts
    16,948
    Thanks
    5
    Quote Originally Posted by danc_ie View Post
    Cool, thanks, that is just what I was looking for.
    I'd like to make a citation to support this - would
    you happen to know of a specific text I could use as
    a reference?

    Cheers,
    dan
    Try reading this first: http://kochanski.org/gpk/teaching/04...confidence.pdf
    Follow Math Help Forum on Facebook and Google+

  5. #5
    Newbie
    Joined
    Oct 2008
    Posts
    4

    Smile Very informative, thanks.

    Quote Originally Posted by mr fantastic View Post
    Although I was hoping for something which had been peer-reviewed,
    that was a very useful resource nonetheless.

    Cheers for your help Mr. Fantastic.

    dan
    Follow Math Help Forum on Facebook and Google+

  6. #6
    Flow Master
    mr fantastic's Avatar
    Joined
    Dec 2007
    From
    Zeitgeist
    Posts
    16,948
    Thanks
    5
    Quote Originally Posted by danc_ie View Post
    Although I was hoping for something which had been peer-reviewed,
    that was a very useful resource nonetheless.

    Cheers for your help Mr. Fantastic.

    dan
    Most statistics textbooks will have the same thing.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Statistical Significance
    Posted in the Statistics Forum
    Replies: 1
    Last Post: November 14th 2009, 03:50 PM
  2. Statistical Significance Problem
    Posted in the Statistics Forum
    Replies: 0
    Last Post: October 6th 2009, 01:12 PM
  3. help on Confidence Interval statistical question
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: May 7th 2009, 07:48 AM
  4. Statistical Significance
    Posted in the Advanced Statistics Forum
    Replies: 1
    Last Post: August 4th 2008, 02:07 AM
  5. Calculating Statistical Significance?
    Posted in the Advanced Statistics Forum
    Replies: 12
    Last Post: June 29th 2006, 10:59 PM

Search Tags


/mathhelpforum @mathhelpforum