minimum required sample size

Jan 2009
47
3
Can I use the following formula in Excel to calculate the required sample size for some A/B testing for various differing groups?

a = Base lines response rate
b = uplift in base line response rate

minimum required sample size = 26*POWER(SQRT(a*(1-a))/(a*b),2)


source: https://vwo.com/blog/ab-test-duration-calculator/
 
Oct 2012
751
212
Ireland
Can you give us some more information on how you are testing the response rate?
From what I gather your experiment is done so that you test for the normal response rate (which you call "a"). Then you vary something and test for the new response rate.

When you say that b is the "uplift in base line response rate" is that the increase you observed? You're making b the increase rather than the value it increased to?

Like your source says:
When someone asks how long should s/he run an A/B test, the ideal answer would be until eternity or till the time you get results (whichever is sooner). In an A/B test, you can never say with full confidence that you will get statistically significant results after running the test X number of days. Instead, what you can say is that there is 80% (or 95%, whatever you choose) probability of getting statistically significant result (if it indeed exists) after X number of days. But, of course, it may be the case that there is in fact no difference in performance of control and variation so no matter how long you wait, you will never get a statistically significant result.
The figure 26 in your excel formula is probably relates to whether you pick a 80% (or 95%, whatever you choose) probability of getting statistically significant result.
What percentage are you choosing?
 
Feb 2015
2,255
510
Ottawa Ontario
Can I use the following formula in Excel to calculate the required sample size for some A/B testing for various differing groups?
Are you only asking if Excel will handle that?
If so, can you not consult the Excel manual?