confidence that values are different

• Jan 1st 2013, 05:59 AM
jenny1988
confidence that values are different
Hi there,

This isn't a homework question. I'm a PhD student, and I'm currently working on an area that would be a little lengthy and unnecessary to explain here, but I have summarised in the following analogy. Stats is not my strong point, so I would really appreciate it if someone could point me in the right direction:

Imagine that you had a whole bunch of red balls and green balls. You suspect that the red balls may have a very slightly different diameter than the green balls.

If you measure one of each, you would get a value for the diameter of each colour ball and an associated uncertainty with each measurement. For example:

D(red) = 10.5 +/- 0.5
D(green) = 10.6 +/- 0.5

With these values, you can't really decide whether the balls DO have a different diameter or if they are the same.

Now if you measure more and more balls of each colour, then the uncertainty of the diameter with become smaller and smaller. After say 100 measurements of the diameter of each colour of ball, we might have:

mean D(red) = 10.51 +/- 0.05
mean D(green) = 10.59 +/- 0.05

My question is - how many measurements might we like to take until we can say with some degree of confidence that the red balls do (or don't) have a different diameter to the green balls. Obviously it depends to some extent how far apart the mean of the diameters are. If you had:

mean D(red) = 10.5 +/- 0.5
mean D(green) = 20.6 +/- 0.5

Then you can be pretty sure they have different diameters!

I guess in summary, I want to say if:

mean D(red) = X(R) +/- Y(R)
mean D(green) = X(G) +/- Y(G)

Then with what confidence can I say that the diameters are the same or different.

Any help at all with this would be gratefully received! Thank you in advance.
• Jan 1st 2013, 07:45 AM
ILikeSerena
Re: confidence that values are different
Hi jenny1988! :)

What you are describing is a t-test.
More specifically, an independent two-sample t-test with presumably equal variances.

As an example, after 100 measurements of red balls, and 100 measurements of green balls, your t-score is:

$t = \frac {X(R) - X(G)} {\sqrt{Y(R)^2+Y(G)^2} / \sqrt{100}}$

This assumes you have properly determined Y(R) and Y(G).

Then you would have to look it up in a t-test table, specifying df=2x100-2=198.
This will give you the confidence level you ask for.

The proper procedure is:
1. Calculate the t-score with the given formula (see here).
2. Look up p-value that corresponds with this t-score.
You will need to specify the degrees of freedom (df) as given in the article.
You can do this with a graphical calculator, SPSS, Excel, an online calculator, or whatnot.
3. If this p-value is smaller than 0.05, you can say with a confidence of 95% that the diameters are different.
Or with an even smaller p-value, you can say with a confidence of (1-2p)x100% that the diameters are different.
• Jan 1st 2013, 09:23 AM
jenny1988
Re: confidence that values are different
That's awesome! Thank you so much for your help!
• Jan 1st 2013, 09:30 AM
ILikeSerena
Re: confidence that values are different
Just to be safe, how are you getting Y(R) and Y(G)?
Since, if you use the wrong values for those, the result will be entirely off.
• Jan 1st 2013, 10:32 AM
jenny1988
Re: confidence that values are different
Well there's quite a lot of error propagation calculations in my actual work before I arrive at the final uncertainty in the "diameters" - but I was careful and I'm pretty sure that my uncertainties are correct. Famous last words :-)
• Jan 1st 2013, 10:39 AM
ILikeSerena
Re: confidence that values are different
My point is that you probably already reduced Y(R) and Y(G) by a factor $\sqrt{100}$.
Obviously you should not do that twice.

And actually, Y(R) and Y(G) should be calculated with the formula for the standard deviation from the respective sets of measurements.
This may well be larger than the (reduced or not) measurement precision.
Not sure if you are aware of that.
• Jan 1st 2013, 12:00 PM
jenny1988
Re: confidence that values are different
Ah I see. Well now that you've put a name to my problem, I can research how to do the test properly and I will carefully check that I'm doing everything correctly. Thank you again for the advice and for the link - it's saved me lots of time!
• Jan 1st 2013, 12:30 PM
jenny1988
Re: confidence that values are different
Okay - so I see what you mean about using the factor of sqrt(100) twice.

The problem is that I can't calculate the standard deviation using the set of measurements, because I don't actually have any measurements (and it's not possible to make any at this stage). To continue my analogy, imagine that the uncertainty in one measurement of the diameter of a ball is known from previous work - but I extrapolate this idea to imagine putting 100 balls end-to-end to, measuring this distance and, therefore, reduce this uncertainty, right? So I have an uncertainty value that has not come from multiple real data.

With my reduced uncertainty, am I now not able to use the t test as quoted above, because as you said, the n = 100 has already played a part is getting my uncertainty?

I should also say that although I have no measurements in the work I am doing, I do have something I can use as the mean, because I am inputting some values that I think would be roughly the mean I expect and then going through this analysis to find out how small the uncertainties would need to be to say that the measurements are difference with say 95% confidence.

Hope this makes some sense! Thank you.

(edit: as I look at it more though, doesn't the uncertainty (sigma) have n already ingrained into it, even if you do calculate it from real data?)
• Jan 1st 2013, 01:15 PM
ILikeSerena
Re: confidence that values are different
If you already know the uncertainty, you should use that value.
Suppose the uncertainty of a single measurement is $\sigma$ (this is the symbol typically used for the standard deviation of the population as opposed to the sample).
And suppose this is the uncertainty for the red balls as well as the green balls.
Then the uncertainty of the mean of 100 measurements is $\sigma / \sqrt{100}$.
This is called the standard error (SE) as opposed to the standard deviation of a single measurement (denoted $\sigma$ or $s$ depending on its origin).

It this is the case, you should use a z-test.

$z = \frac {X(R) - X(G)}{\sigma / \sqrt{100}}$

If this z-value is greater than 1.96 or less than -1.96, you have a confidence level of 95% that the diameters are different.

Just to make sure, I would also calculate the unreduced standard deviations of your samples, to see if they are in the same range.
Just to avoid nasty surprises.
• Jan 1st 2013, 01:23 PM
ILikeSerena
Re: confidence that values are different
Quote:

Originally Posted by jenny1988
(edit: as I look at it more though, doesn't the uncertainty (sigma) have n already ingrained into it, even if you do calculate it from real data?)

A couple of uncertainties are distinguished.

$\sigma$ - The standard deviation of the population.
This one is usually unknown.
If it is known, perhaps from very reliable earlier measurements, you should do the z-test.
As you can see, n, the number of measurements in a sample, is not ingrained in it.

$s$ - The standard deviation of a sample of n measurements.
The formula is $s=\sqrt{\frac {\sum (x_k-\bar x)^2}{n - 1}}$.
This does indeed contain n, the number of measurements.
I suspect this is the one you need to use.

$SE$ - The standard error, which is the standard deviation of the mean of a sample of n measurements.
The formula is $SE=\frac {\sigma}{\sqrt n}$ or $SE=\frac {s}{\sqrt n}$ depending on whether you know $\sigma$ or not.
• Jan 1st 2013, 01:41 PM
ILikeSerena
Re: confidence that values are different
Let me put it differently, since I suspect I am confusing you.

If you have:

mean D(red) = 10.51 +/- 0.05
mean D(green) = 10.59 +/- 0.05

where these numbers and uncertainties are the result of 100 measurements of each.

Then the corresponding z-value is:

$z = \frac {10.59 - 10.51}{0.05} = 1.6$

Since this is less than 1.96, we have not enough evidence to be able to say that the diameters are different.
So either they are not, or you need more measurements.

The corresponding p-value is:

$p = 0.0548$

It means that, based on the sample, the chance that the diameters are actually different is (1-2p)x100% = 89.04%.
However, this is usually considered insufficient to be able to say they are actually different.
• Jan 1st 2013, 02:01 PM
jenny1988
Re: confidence that values are different
Fantastic - that's all really clear thank you!
• Jan 1st 2013, 02:10 PM
ILikeSerena
Re: confidence that values are different
Erm... just edited the numbers in my previous post, since I made a calculation error.

Quote:

Originally Posted by jenny1988
Fantastic - that's all really clear thank you!

You're welcome!
(Btw, did you notice the "Thanks" hyperlink at the right bottom in the posts?)