In order to ensure accurate and reliable results from A/B testing experiments, it is important to use the right sampling approach and understand the implications of different samples. Statistical sampling is the process of selecting a subset of data from a larger population of data in order to make inferences about the population. In the context of A/B testing, sampling involves randomly selecting visitors from the larger population and directing them to the different variations of a test. This allows for an unbiased comparison of the different variations, as well as a more accurate assessment of the impact of the changes.
Once the sampling approach has been determined, it's important to monitor the progress of the experiment and keep an eye out for any potential sample ratio mismatches (SRMs).
Sample ratio mismatch
Sample ratio mismatch is an issue that can occur in A/B testing experiments, where the expected traffic allocation between variations does not match the observed visitor numbers. This mismatch can be caused by several different factors, including technical issues such as redirection tests with poorly connected visitors, complex tests with JavaScript code that crashes the browser, and even when the URL to the variation is shared via mail or social media.
Detecting SRM
In order to detect SRM, a threshold of 0.999 is used to ensure that false positives are avoided. When an SRM problem is detected, the confidence interval should also be taken into account to understand the size of the problem. If the conversion gain is bigger than the measured SRM, this means that the bias cannot be responsible for all the observed gain, and the conclusion of the experiment can be kept.
The impact of SRM depends on the size of the difference between the expected ratio and the observed ratio, as well as the size of the conversion gain. When an SRM problem is detected, it's important to understand the size of the issue and the cause of the problem, in order to be able to correct the issue before restarting the experiment.
SRM Calculator
In order to help A/B testers with their experiments, AB TASTY provides an online service for SRM analysis. This service can help identify potential SRM issues so that they can be corrected before the experiment is restarted.
Overall, SRM can significantly affect the results of an A/B test experiment, and it's important to take steps to identify potential SRM issues and take corrective action before starting or restarting the experiment.