Gain Probability enables you to ascertain the odds of a strictly positive gain on a variation compared to the original version. It is expressed as a % and is available for the Visitor view (and will be available for the Sessions view soon), for any selected goal. This measurement is visible in the Analysis tab of the new reporting function.
We recommend following three business rules before making a decision after running a test:
The Gain Probability is a statistical index that indicates the odds (as a percentage) of recording a strictly positive improvement for a test goal. This measurement is based on the number of collected conversions.
The Gain Probability enables you to determine the risk percentage (100% minus the Gain Probability).
The Gain Probability is currently available for the Visitor view and will be available for the Sessions view soon. The reading mode is the same but the algorithm varies as a function of the selected view.
If the Gain Probability is equal to or higher than 95%, this means that the collected statistics are reliable and that the variation may be implemented with a low risk factor (5% or less).
By default, AB Tasty displays the Gain Probability of variation 1. You can change the variation linked to the Gain Probability by selecting the desired variation from the drop-down list in the Summary tab.
You can also change the reference version (the original version is selected by default) in the top right-hand section of the page. This version serves as a reference for all displayed calculations.
This enables you to view the Gain Probability of a variation (selected in the list) compared to a chosen reference version.
Gain Probability enables fast result analysis for non-experts. The variation with the highest gain probability is displayed automatically, which facilitates the decision-making process.
This index assists with the decision-making process but we recommend reading the Gain Probability in addition to the confidence intervals (or Bayesian test), which may display positive or negative values.
The Gain Probability calculation varies as a function of the selected view.
In the Visitor view, the calculation is based on the confidence intervals of the Bayesian test.
The index is found in the Summary tab of a goal, which is displayed by default.We recommend viewing the advanced statistics in addition to gain probability.
Sessions view (coming soon)
For the Sessions view, the principle remains the same but the calculation is based on the Mann Whitney U test.
The Gain Probability is the only indicator because the Bayesian test cannot be calculated in the Sessions view. The decision-making process is therefore based purely on this indicator.
The Bonferroni correction is a method that involves taking into account the risk linked to the presence of several comparisons/variations.
In the case of an A/B test, if there are only two variations (the original and variation 1), it is estimated that the winning variation may be implemented if the Gain Probability is equal to or higher than 95%. In other words, the risk incurred does not exceed 5%.
In the case of an A/B test with two or more variations (the original version, variation 1, variation 2 and variation 3, for instance), if one of the variations (let’s say variation 1) performs better than the others and you decide to implement it, this means you are favoring this variation over the original version, as well as over variation 2 and variation 3. In this case, the risk of loss is multiplied by 3 (5% multiplied by the number of “abandoned” variations).
A correction is therefore automatically applied to tests featuring one or more variations. Indeed, the displayed Gain Probability takes the risk related to abandoning the other variations into account. This enables the user to make an informed decision with full knowledge of the risks related to implementing a variation.
When the Bonferroni correction is applied, there may be inconsistencies between the Gain Probability and the confidence interval displayed in the Advanced tab. This is due to the fact that the Bonferroni correction does not apply to the Bayesian test.
The Gain Probability can take values between 0% and 100% and is rounded to the nearest hundred.
- The closer the value is to 0%, the higher the odds of it underperforming compared to the original version and the higher the odds of having confidence intervals with negative values.
- At 50%, the test is considered as neutral. There is a much chance of the variation underperforming compared to the original variation as there is of it overperforming. The confidence intervals can take negative or positive values. The test is either neutral or has not enough data.
- The closer the value is to 100%, the higher the odds of recording a gain compared to the original version. The confidence intervals are more likely to take on positive values.
If the Gain Probability displays 0% or 100% in the reporting tool, these figures are rounded up/down. A statistical probability can never equal exactly 100% or 0%. It is therefore preferable to display 100% rather than 99,999999% to facilitate report reading for users.
Case #1: High Gain Probability
In this example, the chosen goal is the revisit rate in Visitor view. The A/B test includes three variations.
The conversion rate of variation 3 is 29.9%, compared to 29.19% for the original version. Therefore, the increase in conversion rate compared to the original equals 2.4%.
The Gain Probability displays 98.23% for variation 3 (the Bonferroni correction is applied automatically because the test includes three variations). This means that variation 3 has a 98.23% chance of triggering a positive gain, and therefore of performing better than the original version. The odds of this variation performing worse than the original therefore equal 1.8%, which is a low risk.
Because the Gain Probability is higher than 95%, variation 3 may be implemented without incurring a high risk.
However, in order to find out the gain interval and reduce the risk percentage even more, we would need to also analyze the advanced statistics based on the Bayesian test.
Case #2: Neutral Gain Probability
If the test displays a Gain Probability around 50% (between 45% and 55%), this can be due to several factors:
- Either traffic is insufficient (in other words, there haven't been enough visits to the website and the visitor statistics do not enable us to establish reliable values): in this case, we recommend waiting until each variation has clocked 5,000 visitors and a minimum of 500 conversions.
- Or the test is neutral because the variations haven't shown an increase or a decrease compared to the original version: this means that the tested hypotheses have no effect on the conversion rate.
In this case, we recommend referring to the Bayesian test in the Advanced tab. This will provide you with the confidence interval values.
If the Bayesian test does not enable you to ascertain a clear gain, the decision will have to be made independently from the test, based on external factors (such as implementation cost, development time, etc.)
Case #3: Low Gain Probability
In this example, the chosen goal is the CTA click rate in Visitor view. The A/B test is made up of a single variation.
The conversion rate of variation 1 is 14.76%, compared to 15.66% for the original version. Therefore, the conversion rate of variation 1 is 5.75% lower than the original version.
The Gain Probability displays 34.6% for variation 1. This means that variation 1 has a 34.6% chance of triggering a positive gain, and therefore of performing better than the original version. The odds of this variation performing worse than the original therefore equal 65.4%, which is a very high risk.
Because the Gain Probability is lower than 95%, variation 1 should not be implemented: the risk would be too high.
In this case, you can view the advanced statistics in order to make sure the confidence interval values are mostly negative.