Carter Gilchrist
posted this on October 19, 2010 01:44 pm
When you are running a split test between multiple variants (we'll just say two in this case; A and B) you will need to assign a weight to each. This weight is the percentage of your overall traffic that you'd like to see either A or B.
Now, every time a new visitor comes to your page, we'll randomly serve up a variant based on the weighting you have specified. If you've assigned both A and B a 50% weight, then this will be no different than a coin toss for each page request.
Something to note is that some imbalances may occur while the traffic to a page is still low. So, as in flipping a coin, you may get heads 5 times in a row.
We should also note that when a visitor first accesses your landing page, we deposit a cookie on that visitor's machine to track the view and to ensure that they see the same variant each time they return. If you visit your own page's URL, you might notice that you continually see the same variant no matter how many times you return or refresh your browser--this is because of the tracking cookie. To workaround this, just clear your browser's cache and try again but keep in mind that you will then be counted as another 'new' unique visitor.
Comments
Hey Carter, What do you call "low" traffic to a page? I recently ran a test with over 650 visitors to a page, 4 variants each at 25% weight, but there were significant differences in the number of visitors per page (+/- 20 visitors per variant)... this seems fairly inaccurate weighting for so many visits? Thanks!
Hey James, sorry for the delay -- I wanted to get a second opinion before responding to you. The difference in visitors that you're seeing is actually not all that irregular or inaccurate, and it won't affect the statistical validity of your test in any way. Have you allowed that test to reach statistical significance yet (where the confidence interval is 95% or higher for some of the variants)?