Comparing Response Rates

Printable View

December 22nd 2008, 01:47 AM
lucas456

Comparing Response Rates

I have 2 adverts running.

The first advert gets 203 views and 4 people respond. That is 1.97% respond.

The second advert gets 191 views and just 1 person responds. That is 0.52% people respond.

How confident can you be that the 2 adverts will remain to have significantly different response rates in the future?

I have spent probably 4 hours searching the net and looking in my stats book for an answer to this. If anyone could explain how to do this in simple english that would be greatly appreciated.
December 24th 2008, 05:04 AM
Pim

The distribution that fits with this problem is a binomial one, you get a visitor (trial), who will (or will not) click your ad. (Success/failure)
There is a formula, which can give you confidence intervals for proportions:

$ P \pm z_c \sqrt{\frac{pq}{n}} $

With P as the sample p, $z_c$ as your confidence level, p as the actual success, which in this case will be approximated by P, q as p -1, failure chance and n as the sample size.

We'll take 95% as your confidence level, which converts to $z_c = 1.96$

Two times computing:

$p_{actual}$ being the true p

$ n = 203 $

$ p = 0.0197 $
$0.0197 \pm 1.96 \sqrt{\frac{0.0197 * 0.9803}{203}}$

$0.0197 \pm 0.01911704079$

$0.0005829592104 \leq p_{actual} \leq 0.03881704079$

$ n = 191 $

$ p = 0.0052 $
$0.0052 \pm 1.96 \sqrt{\frac{0.0052*0.9948}{191}}$

$0.0052 \pm 0.0102002061$

$0 \leq p_{actual} \leq 0.01540020612$

(The first part would be smaller than 0, but per defenition, chances cannot be smaller than zero)

Now, as you can see those intervals overlap for quite a large part, however they are indeed significantly different. What this means is that it's not unlikely for them to remain different. However, if you want more certainity, I suggest you compute this again once you have quite a bit more views, as that means there are more trials and therefore your intervals will get shorter and you will see how much so.
December 25th 2008, 01:33 PM
lucas456

Thank you so much!

Thank you so much!

I was beginning to think I'd be stuck forever after asking several forums!

I am humbled by your reply, but would you be so kind as to define how far apart these figures have to be, to be significantly different? Or can you even define it?

Sorry if I'm asking a stupid question and thanks again!
December 25th 2008, 11:16 PM
Pim

The problem of this lies in mainly this: How significantly different do you need them to be? What this formula does, it gives you intervals for which you can be sure that the average response rate will stay in, with 95% certainity. Now if these intervals do not overlap, they will most likely not get the same any time soon. That is what I would call "significantly different". I hope that you understand how to fill in this formula and I suggest you just try it again at a thousand or so views. Another thing you could do, is compute them again with another certainity rate ( $z_c$ ). Here are a few values for which you can be x% certain that the mean will stay within the interval you get as a result:
50% -- 0.6745
80% -- 1.28
90% -- 1.645
December 26th 2008, 06:31 AM
lucas456

Quote:

Originally Posted by Pim

The problem of this lies in mainly this: How significantly different do you need them to be? What this formula does, it gives you intervals for which you can be sure that the average response rate will stay in, with 95% certainity. Now if these intervals do not overlap, they will most likely not get the same any time soon. That is what I would call "significantly different". I hope that you understand how to fill in this formula and I suggest you just try it again at a thousand or so views. Another thing you could do, is compute them again with another certainity rate ( $z_c$ ). Here are a few values for which you can be x% certain that the mean will stay within the interval you get as a result:
50% -- 0.6745
80% -- 1.28
90% -- 1.645

Thanks for your help and yes, I do understand how to fill in the formula with the correct values.
December 26th 2008, 09:15 AM
Constatine11

Quote:

Originally Posted by Pim

The distribution that fits with this problem is a binomial one, you get a visitor (trial), who will (or will not) click your ad. (Success/failure)
There is a formula, which can give you confidence intervals for proportions:

$ P \pm z_c \sqrt{\frac{pq}{n}} $

With P as the sample p, $z_c$ as your confidence level, p as the actual success, which in this case will be approximated by P, q as p -1, failure chance and n as the sample size.

We'll take 95% as your confidence level, which converts to $z_c = 1.96$

Two times computing:

$p_{actual}$ being the true p

$ n = 203 $

$ p = 0.0197 $
$0.0197 \pm 1.96 \sqrt{\frac{0.0197 * 0.9803}{203}}$

$0.0197 \pm 0.01911704079$

$0.0005829592104 \leq p_{actual} \leq 0.03881704079$

$ n = 191 $

$ p = 0.0052 $
$0.0052 \pm 1.96 \sqrt{\frac{0.0052*0.9948}{191}}$

$0.0052 \pm 0.0102002061$

$0 \leq p_{actual} \leq 0.01540020612$

(The first part would be smaller than 0, but per defenition, chances cannot be smaller than zero)

Now, as you can see those intervals overlap for quite a large part, however they are indeed significantly different. What this means is that it's not unlikely for them to remain different. However, if you want more certainity, I suggest you compute this again once you have quite a bit more views, as that means there are more trials and therefore your intervals will get shorter and you will see how much so.

The normal approximations is not valid in this case. What I would suggest is that you take the null hypothesis that there is no difference in the proportions and that the common proportion is about 0.0127, and bootstrap a 95% interval for the difference in the proportions for samples of size 203 and 191.

Doing this with a bootstrap sample of 50000 gives a 95% interval of about [-0.022,0.024] for the difference in proportions between the two samples.

As the observed difference is 0.0145 and is well within this interval there is no strong evidence that the response rate is significantly differeent for the two ads. (in fact the observed difference is well inside the 90% interval and only just outside the 75% interval)