# Thread: Claiming statistical significance - P-value or confidence interval?

1. ## Claiming statistical significance - P-value or confidence interval?

Greetings all,

I am testing a modification to an algorithm to see if significant
performance gains can be observed when compared with the
original algorithm.

I have performed 30 trials of each on a benchmark problem and I
have two sets of performance results to compare (higher values
being better).

Using a 95% confidence interval, I can observe (graphically) that
there is no overlap between the error-bars, the modified method
shows mean performance window "above" the original method.

Using a paired t-test, I obtain a P-value less than 0.05.

My question is -- are these two methods functionally equivalent?
In a paper I'm writing I am using the confidence intervals to claim
a statistically significant performance improvement.

Is this sufficient or am I potentially leaving myself open to criticism
by more experienced statisticians than myself?

Any comments would be most welcome. Many thanks,

dan

2. Originally Posted by danc_ie
Greetings all,

I am testing a modification to an algorithm to see if significant
performance gains can be observed when compared with the
original algorithm.

I have performed 30 trials of each on a benchmark problem and I
have two sets of performance results to compare (higher values
being better).

Using a 95% confidence interval, I can observe (graphically) that
there is no overlap between the error-bars, the modified method
shows mean performance window "above" the original method.

Using a paired t-test, I obtain a P-value less than 0.05.

My question is -- are these two methods functionally equivalent?
In a paper I'm writing I am using the confidence intervals to claim
a statistically significant performance improvement.

Is this sufficient or am I potentially leaving myself open to criticism
by more experienced statisticians than myself?

Any comments would be most welcome. Many thanks,

dan
Finding a confidence interval or a P-value are equivalent ways of doing statistical inference at a given significance level. Be sure to quote the P-value you got, don't just say it's less than 0.05.

3. ## Any references?

Originally Posted by mr fantastic
Finding a confidence interval or a P-value are equivalent ways of doing statistical inference at a given significance level.
Cool, thanks, that is just what I was looking for.
I'd like to make a citation to support this - would
you happen to know of a specific text I could use as
a reference?

Cheers,
dan

4. Originally Posted by danc_ie
Cool, thanks, that is just what I was looking for.
I'd like to make a citation to support this - would
you happen to know of a specific text I could use as
a reference?

Cheers,
dan

5. ## Very informative, thanks.

Originally Posted by mr fantastic
Although I was hoping for something which had been peer-reviewed,
that was a very useful resource nonetheless.

Cheers for your help Mr. Fantastic.

dan

6. Originally Posted by danc_ie
Although I was hoping for something which had been peer-reviewed,
that was a very useful resource nonetheless.

Cheers for your help Mr. Fantastic.

dan
Most statistics textbooks will have the same thing.