Comments on: Determining your Sample Size http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/ Web Analytics Blog - Paving the way to understanding web data as it relates to statistics and other methodologies. Thu, 28 Aug 2008 15:58:37 +0000 http://wordpress.org/?v=2.1.3 By: Steve http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-25 Steve Fri, 06 Jul 2007 02:26:17 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-25 Thanks Wendi, at this rate I may as well just automatically "star" all your postings in Google Reader... ;-) I would interpret this to mean that Robbin possibly does have enough info to make a determination (3 x ~ 250 + other combos?). Does that work? Or should we have 1700 for those three only? Assuing that to be the case. Should she stick with the original or??? I see myself torn between the original and #1. #1 Reduces the error range - not by much. That seems to imply to me that it is a more guaranteed combination? And hence has a value in and of itself??? Or am I dreaming? :-) I could always handle the math side of statistics (I can just about read ancient greek purely from studying engineering @ uni. ;-) ). Knowing how/when to apply it was ever my problem... Cheers, and Thanks! Thanks Wendi, at this rate I may as well just automatically “star” all your postings in Google Reader… ;-)

I would interpret this to mean that Robbin possibly does have enough info to make a determination (3 x ~ 250 + other combos?). Does that work? Or should we have 1700 for those three only?

Assuing that to be the case. Should she stick with the original or???

I see myself torn between the original and #1.
#1 Reduces the error range - not by much. That seems to imply to me that it is a more guaranteed combination? And hence has a value in and of itself???

Or am I dreaming? :-)

I could always handle the math side of statistics (I can just about read ancient greek purely from studying engineering @ uni. ;-) ). Knowing how/when to apply it was ever my problem…

Cheers, and Thanks!

]]> By: Wendi http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-26 Wendi Fri, 06 Jul 2007 15:23:10 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-26 Hi Steve, Thanks for the positive feedback and glad you find this blog useful. Actually Robbin needs a full 1700 with those three combinations. I don't believe that she was running any others. But yes, she just needs a total of 1700 for the entire sample. So if she did run other combinations then she would have been closer to what she needed. As for picking the right one in the end, well that is up to her since there didn't seem to be any impact from the treatments. It would be more of a personal preference question at that point. Maybe over time the conversion will experience a lift just by refreshing the page anyway - who knows. But right now if it takes an act of congress to update the page, I would waiver against not making any changes. In some organizations there tends to be this struggle for IT resources so if that exists, based on the tested outcomes she may not be wise to make any changes at this point. But really we are too early in the game to make a sound decision. I guess it's just a waiting game at this point. Take Care, Wendi Hi Steve, Thanks for the positive feedback and glad you find this blog useful.
Actually Robbin needs a full 1700 with those three combinations. I don’t believe that she was running any others. But yes, she just needs a total of 1700 for the entire sample. So if she did run other combinations then she would have been closer to what she needed.
As for picking the right one in the end, well that is up to her since there didn’t seem to be any impact from the treatments. It would be more of a personal preference question at that point. Maybe over time the conversion will experience a lift just by refreshing the page anyway - who knows. But right now if it takes an act of congress to update the page, I would waiver against not making any changes. In some organizations there tends to be this struggle for IT resources so if that exists, based on the tested outcomes she may not be wise to make any changes at this point. But really we are too early in the game to make a sound decision. I guess it’s just a waiting game at this point.

Take Care, Wendi

]]> By: Steve http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-30 Steve Mon, 09 Jul 2007 09:05:47 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-30 Hi Wendi, thanks again (very muchly!) for that. Can I please query one more issue (hopefully... ;-) ) You've set the margin of error to 0.01. I gather from reading elsewhere that implies an margin of error of 1%. If we increase that error to 1.5% we drop the number of pages required to 768. Which she already has (783). So the question: Why would we continue to 1% when by accepting just an extra 1/2 a percent, we're already there? Obviously office/client politics, but are there formulaic reasons why that would be preferable? Is it tied to the Z-value in some way? If I may be allowed a bonus question? :-) To try and put it terms I am familiar with: Are we already at a level of confidence of 95% +/- 0.32%? Or have I totally got it all wrong? (again) I think I'm actually starting to understand this stuff... :-) Cheers! and again, Thanks! Hi Wendi, thanks again (very muchly!) for that.

Can I please query one more issue (hopefully… ;-) )
You’ve set the margin of error to 0.01. I gather from reading elsewhere that implies an margin of error of 1%.
If we increase that error to 1.5% we drop the number of pages required to 768. Which she already has (783).

So the question: Why would we continue to 1% when by accepting just an extra 1/2 a percent, we’re already there? Obviously office/client politics, but are there formulaic reasons why that would be preferable? Is it tied to the Z-value in some way?

If I may be allowed a bonus question? :-)
To try and put it terms I am familiar with: Are we already at a level of confidence of 95% +/- 0.32%?
Or have I totally got it all wrong? (again)

I think I’m actually starting to understand this stuff… :-)

Cheers! and again, Thanks!

]]> By: Robbin Steif http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-33 Robbin Steif Tue, 10 Jul 2007 02:09:59 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-33 Wendi - is the "margin of error" a margin of the control, i.e. 1% of the 4.72% conversion rate, or additive, 5.72%? I am guessing the former, since the latter seems too detectable to be considered a tie. Wendi - is the “margin of error” a margin of the control, i.e. 1% of the 4.72% conversion rate, or additive, 5.72%? I am guessing the former, since the latter seems too detectable to be considered a tie.

]]> By: Wendi http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-35 Wendi Tue, 10 Jul 2007 04:17:25 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-35 Hi Steve - Yes, by toggling the width of your precision you can reduce the amount of work needed to accomplish your goal. It just depends on how much you are willing to give up in your precision. Remember that the margin of error is plus or minus so the more you give the less precision you have. And yes, you are almost correct in your margin of error estimate for the current conditions. It's actually +/- 3.2% not 0.32%. In many cases you will see polls refer to the standard +/- 3% error rate and right now Robbin is pretty much inline with the standard needs of acceptability. Hope this helps. Wendi Hi Steve - Yes, by toggling the width of your precision you can reduce the amount of work needed to accomplish your goal. It just depends on how much you are willing to give up in your precision. Remember that the margin of error is plus or minus so the more you give the less precision you have. And yes, you are almost correct in your margin of error estimate for the current conditions. It’s actually +/- 3.2% not 0.32%. In many cases you will see polls refer to the standard +/- 3% error rate and right now Robbin is pretty much inline with the standard needs of acceptability.

Hope this helps.
Wendi

]]> By: Wendi http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-36 Wendi Tue, 10 Jul 2007 04:26:45 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-36 Robbin - The margin of error is the window that the true conversion rate should fall - with 95% confidence. In other words, you are 95% confident that the true (population) conversion rate is somewhere between [3.72%,5.72%] based on your sampling (test). Cheers! Wendi Robbin - The margin of error is the window that the true conversion rate should fall - with 95% confidence. In other words, you are 95% confident that the true (population) conversion rate is somewhere between [3.72%,5.72%] based on your sampling (test).

Cheers! Wendi

]]> By: Wendi http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-38 Wendi Tue, 10 Jul 2007 13:29:06 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-38 All Readers - I made a booboo - I mis-typed the the formula above. I forgot to include the squaring of the delta. The excel file that is correct and the actual value calculated and originally posted was also correct. The post has been updated and is now correct. Sorry for this. Thanks Robbin!! All Readers - I made a booboo - I mis-typed the the formula above. I forgot to include the squaring of the delta. The excel file that is correct and the actual value calculated and originally posted was also correct. The post has been updated and is now correct. Sorry for this.

Thanks Robbin!!

]]> By: Michael Helbling http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-46 Michael Helbling Sat, 14 Jul 2007 21:29:54 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-46 I am really enjoying reading your posts. This one was excellent, but I have one issue. You mention that the number of page views should be 1,728. In reality shouldn't it be 1,728 visits or sessions that enter on this page? Since a single visit could realistically see that page 2-4 times, I would think you would only want to count the first page view in a session toward the sample size. I am really enjoying reading your posts. This one was excellent, but I have one issue. You mention that the number of page views should be 1,728. In reality shouldn’t it be 1,728 visits or sessions that enter on this page? Since a single visit could realistically see that page 2-4 times, I would think you would only want to count the first page view in a session toward the sample size.

]]> By: Steve http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-48 Steve Sun, 15 Jul 2007 22:56:11 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-48 Apologies for late reply. Thanks Wendi. I did wonder about the 0.32 - it felt wrong. :-) Helps hugely! Again (x 2, x3 , x4 ...) thanks! Cheers! Apologies for late reply. Thanks Wendi. I did wonder about the 0.32 - it felt wrong. :-)

Helps hugely! Again (x 2, x3 , x4 …) thanks!
Cheers!

]]> By: Wendi http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-49 Wendi Mon, 16 Jul 2007 13:46:02 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-49 Hi Michael, Very good point.... but: Since Robin is measuring CTR - Click Thru Rate where CTR = Clicks / Page Views (aka Impressions) Then looking at Page Views makes more sense. Actually in Google Analytics you can pull Unique Page Views which is what you should look at to be most accurate. Robbin could look at the number of reloads and back those out as well. Thanks for the comment. Cheers! Wendi Hi Michael, Very good point…. but:
Since Robin is measuring CTR - Click Thru Rate where
CTR = Clicks / Page Views (aka Impressions)
Then looking at Page Views makes more sense. Actually in Google Analytics you can pull Unique Page Views which is what you should look at to be most accurate. Robbin could look at the number of reloads and back those out as well.

Thanks for the comment.

Cheers!
Wendi

]]> By: chrisg http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-50 chrisg Mon, 16 Jul 2007 19:06:26 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-50 I think Michael's point is that the power calculation formula assumes independence of all of the observations, and two views of the same page by one person are not independent observations. To get the best possible estimate of sample sizes needed, you'd have to use only one data point per visit. Even though your statistic (CTR) is based on page views, your probability stats, which is what your power calculations are based on, have to use no more than one event (page view) per visit. If it were my analysis, I would set it up to use only the first view of the page in a visit, both for the sample size formulas and the estimate of CTR itself. Regards, Chris I think Michael’s point is that the power calculation formula assumes independence of all of the observations, and two views of the same page by one person are not independent observations. To get the best possible estimate of sample sizes needed, you’d have to use only one data point per visit. Even though your statistic (CTR) is based on page views, your probability stats, which is what your power calculations are based on, have to use no more than one event (page view) per visit. If it were my analysis, I would set it up to use only the first view of the page in a visit, both for the sample size formulas and the estimate of CTR itself.

Regards, Chris

]]> By: Wendi http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-51 Wendi Mon, 16 Jul 2007 19:43:26 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-51 Hey Chris. Absolutely. I am following you and Mike. That is why I included the usage of Unique Page Views that Google Analytics provides in my reply to Mike (and is what Robbin is using on their site). I agree totally with both of you. I'll post and update to make that point clear in the post. Thanks again! Wendi Hey Chris. Absolutely. I am following you and Mike. That is why I included the usage of Unique Page Views that Google Analytics provides in my reply to Mike (and is what Robbin is using on their site). I agree totally with both of you. I’ll post and update to make that point clear in the post.

Thanks again!
Wendi

]]> By: Erik http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-279 Erik Tue, 28 Aug 2007 23:49:41 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-279 Thanks Wendi, This has been a really helpful article as I get prepared for a testing program. I have a question regarding the number of variants (or 'combinations' as google calls them). I read the above N as 1728 total views, or an average of 576 views for each of the three variants. What happens if we are testing 6 different variants or even 9 (as in some multivariate testing)? If your N is fixed at 1728, the required views per variant would decrease as variants are added. At some point it seems that you would get too few views per variant to have a statistically significant test. It seems to me the number of variants should be factored into the total sample size. In reading about the Power formula, it seems it is based on a 2 sample test. If that's the case should the above formula be multiplied by (r / 2), where 'r' is the number of variants? Hope I'm on the right track with this, if not please let me know, Thanks! Thanks Wendi, This has been a really helpful article as I get prepared for a testing program. I have a question regarding the number of variants (or ‘combinations’ as google calls them).

I read the above N as 1728 total views, or an average of 576 views for each of the three variants. What happens if we are testing 6 different variants or even 9 (as in some multivariate testing)? If your N is fixed at 1728, the required views per variant would decrease as variants are added. At some point it seems that you would get too few views per variant to have a statistically significant test.

It seems to me the number of variants should be factored into the total sample size. In reading about the Power formula, it seems it is based on a 2 sample test. If that’s the case should the above formula be multiplied by (r / 2), where ‘r’ is the number of variants? Hope I’m on the right track with this, if not please let me know,
Thanks!

]]> By: Rachel http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-405 Rachel Wed, 26 Sep 2007 13:32:27 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-405 I am still a bit confused on the sample size. Does she need 1700 visitors for each experiment or in total for all three? Right now in total she is at 783 and I'm trying to figure out if she needs ~1000 more or 4300 more. Thanks. I am still a bit confused on the sample size. Does she need 1700 visitors for each experiment or in total for all three? Right now in total she is at 783 and I’m trying to figure out if she needs ~1000 more or 4300 more. Thanks.

]]> By: seby kallarakkal http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-1529 seby kallarakkal Tue, 27 Nov 2007 02:49:41 +0000 http://coremarkanalytics.com/blog/2007/07/05/determining-your-sample-size/#comment-1529 Hi Wendi, Great post. Thanks for sharing. I came across your blog few days back and have been catching up on the older posts. Your perspective on using stats for analytics is proving to be really useful. Hi Wendi,
Great post. Thanks for sharing. I came across your blog few days back and have been catching up on the older posts. Your perspective on using stats for analytics is proving to be really useful.

]]>