Comments on: Comparing Population Proportions – A/B Testing http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/ Web Analytics Blog - Paving the way to understanding web data as it relates to statistics and other methodologies. Fri, 30 Jul 2010 08:40:43 +0000 http://wordpress.org/?v=2.1.3 By: Coremark Analytics » Determining your Sample Size http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-24 Coremark Analytics » Determining your Sample Size Thu, 05 Jul 2007 22:59:34 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-24 [...] are statistically significant. You can read more about how to do this on my previous post about A/B testing. You can find a downloadable excel file in this post that you can toggle various sample sizes and [...] […] are statistically significant. You can read more about how to do this on my previous post about A/B testing. You can find a downloadable excel file in this post that you can toggle various sample sizes and […]

]]> By: thierry http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-117 thierry Tue, 31 Jul 2007 14:23:54 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-117 Hi, I was going thru your article, really like it. I think many people will find it useful :-) Just a short question, you use 1 - NORMSDIST(1.2563) and are therefore using 1 tailed test, but in fact it seems that you want to test whether p1 ≠ p2 which would be 2 tailed? i.e. p1 could be higher than p2 but it could also be lower... If that's the case you might consider using 2*NORMSDIST(-ABS(1.2563)) instead? Thank you Thierry Hi,

I was going thru your article, really like it. I think many people will find it useful :-)

Just a short question, you use 1 - NORMSDIST(1.2563) and are therefore using 1 tailed test, but in fact it seems that you want to test whether p1 ≠ p2 which would be 2 tailed? i.e. p1 could be higher than p2 but it could also be lower…

If that’s the case you might consider using 2*NORMSDIST(-ABS(1.2563)) instead?

Thank you

Thierry

]]> By: Wendi http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-118 Wendi Wed, 01 Aug 2007 03:10:25 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-118 Hi Thierry, You are absolutely correct. You can achieve the same calculation as you have mentioned above by just simply doubling 1-NORMSDIST() calculation. Thus 2*[1-NORMSDIST()]. I will make and edit to the above for better clarification. Thanks. Hi Thierry, You are absolutely correct. You can achieve the same calculation as you have mentioned above by just simply doubling 1-NORMSDIST() calculation. Thus 2*[1-NORMSDIST()]. I will make and edit to the above for better clarification. Thanks.

]]> By: Sylvester http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-866 Sylvester Wed, 31 Oct 2007 15:36:44 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-866 Wendi, what happens when you've got three tests running: a control and two variations. I can guess what to do with the denominator in your z1 formula, but not the numerator Wendi,

what happens when you’ve got three tests running: a control and two variations. I can guess what to do with the denominator in your z1 formula, but not the numerator

]]> By: tazman http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-15340 tazman Mon, 30 Mar 2009 22:07:57 +0000 http://coremarkanalytics.com/blog/2007/06/17/comparing-population-proportions-%e2%80%93-ab-testing/#comment-15340 Wendi: Another little 'gotcha' I discovered is that you had to put the higher performing number in the LP1 Conversion cell, otherwise you would always fail to reject! It took me a bit of playing around to figure that out as I couldn't understand how I wasn't getting significant results (and I had sample sizes in the 750k+ range)! A quick fix so you don't have to remember to do that is add an if caluse in the p1-p2 calculation like this: =IF(C7>D7,C7-D7,D7-C7) that way you don't have to worry about the order of entry... Wendi:

Another little ‘gotcha’ I discovered is that you had to put the higher performing number in the LP1 Conversion cell, otherwise you would always fail to reject! It took me a bit of playing around to figure that out as I couldn’t understand how I wasn’t getting significant results (and I had sample sizes in the 750k+ range)! A quick fix so you don’t have to remember to do that is add an if caluse in the p1-p2 calculation like this:

=IF(C7>D7,C7-D7,D7-C7)

that way you don’t have to worry about the order of entry…

]]>