Jul 05 2007
Determining your Sample Size
Robbin Steif asked me today how long she needed to let her test run before she could call it a day and assume that there is really no difference between the treatments (since she isn’t seeing one right now). She sent over the following screen shot of her outcomes in Google’s Website Optimizer from the last two weeks:
As you can see, right now she isn’t seeing any lift in her conversion rate. Actually she is seeing similar values and a small drop. But is the drop significant? Does she have enough data to support an outcome at this point?
Before you run a test of significance you first need to know if you have enough data to support the test in the first place. For population proportions the formula for sample size “n” is:
n = z2(pq/δ2)
where
p = % of Success (conversions in this example)
q = % of Failures (i.e. 1 – p)
*note: use the Conversation Rate from your control landing page
To finish out this equation you need to make a few assumptions.
Assumptions
1. The Confidence Level - α (alpha): the level of certainty that you are willing to accept
2. Error - δ (delta): the margin of error that you are willing to accept
With these assumptions set, lastly you need to calculate the Z value based on your Confidence Level. It’s easy to do in excel with the NORMSINV() formula. Since we are determining the existence of a “difference” among the conversion rates versus if the conversion rate is specifically higher or lower than the control we need to divide alpha in half for a two-sided test structure.
=ABS(NORMSINV(α/2))
In this example our Z = 1.96. Now we have all the pieces in our formula to calculate the needed sample size.
n = z2(pq/δ2)
= (1.96)2 * [(.0472*.9528)/(.01)2]
= 1728
Thus Robbin is going to need 1,728 page views before she can make the determination that the treatments she is testing did or did not make a difference in her conversion rate. You can download this excel file I put together (nothing fancy) where you can toggle the alpha and delta values so that you can see how each one impacts the needed sample size.
I also included a reference to the maximum sample size one would need if you don’t have a control to set your “p” and “q” values. It’s rather astonishing but it you are a conservative then you can always fall back on this calculation and know that if you get approximately 10,000 samples you are good to go.
Part II of this question is – Is there a difference? This is different than asking how many samples you need to determine if there is a difference. Applying a hypothesis test is needed to actually determine if the difference in the conversion rates are statistically significant. You can read more about how to do this on my previous post about A/B testing. You can find a downloadable excel file in this post that you can toggle various sample sizes and determine if the conversion rates are different – statistically speaking.
Until next time… safe analyzing.
*UPDATE* 7/16/07: Make sure that you use the first page view per visitor - “Unique Page Views” in Google analytics Terms when making this calculation. The sample size calculation assumes each event is independent of each other.
Thank You to Mike & Chris!