How can I determine the accuracy of some predictions?

May 2010
1
0
How can I determine the accuracy of some predictions? Take the example of weather forecasts. Each day weather.com has a predicted probability it will rain. Over a week then, we know the predicted
chance it would rain, and whether it rained on each day. How do I determine how accurate the predictions were over the past week?

For instance, for Sunday-Saturday of one week, here's the predictions and the actual results:

Day of week: Predicted Chance of Rain - Actual result (1 = it rained, 0 = it didn't rain)

S: .72 - 1
M: .31 - 0
T: .08 - 0
W: .01 - 0
T: .24 - 1
F: .51 - 1
S: .98 - 1

So given this, how do I determine the accuracy of the prediction percentages?
 

undefined

MHF Hall of Honor
Mar 2010
2,340
821
Chicago
How can I determine the accuracy of some predictions? Take the example of weather forecasts. Each day weather.com has a predicted probability it will rain. Over a week then, we know the predicted
chance it would rain, and whether it rained on each day. How do I determine how accurate the predictions were over the past week?

For instance, for Sunday-Saturday of one week, here's the predictions and the actual results:

Day of week: Predicted Chance of Rain - Actual result (1 = it rained, 0 = it didn't rain)

S: .72 - 1
M: .31 - 0
T: .08 - 0
W: .01 - 0
T: .24 - 1
F: .51 - 1
S: .98 - 1

So given this, how do I determine the accuracy of the prediction percentages?
I can partially answer your question. This is basically a situation for expected value. Assuming the predictions are fairly accurate, we would think that the following holds (but generally for larger sample size)

\(\displaystyle 0.72 + 0.31 + 0.08 + 0.01 + 0.24 + 0.51 + 0.98 \approx \frac{1+0+0+0+1+1+1}{7}\)

I'm guessing that a reasonable way to quantify the error here (which would reflect accuracy/precision) is to take the absolute difference of the LHS and RHS and divide by sample size, but this is a bit of "original thinking" on my part.

It's a good question. I hope that if my answer is inadequate someone will provide a better one.