👉 Dynamic allocation

Dynamic allocation is an option on the Traffic allocation step of all A/B Tests. 

📖 Definition

Dynamic traffic allocation is based on a statistical research topic called “multi-armed bandit”. The goal of this algorithm is to limit loss due to the lowest performing variations. 
However, when the campaign has just been launched, the performance of the variations is not known yet. When data collection begins, the performance of variations is known progressively, but these measures are not fully significant.
At this point, it would be an error to send all traffic to the apparent winner. However, some formulas can help compute the right amount of traffic to send to each variation with the following goal: sending more traffic to the variation that could win, but not too much, otherwise there wouldn’t be enough traffic to ensure the measurement of the supposed losers.
The solution is to periodically change the allocation regarding previous data, following a statistical formula that ensures the goal mentioned above.
Dynamic_allocation.jpg

💡 Use cases

Wait what ? But changing allocation is known as bad practice !!

That is totally true, so there are conditions where you can use dynamic allocation without risk. To understand them you need to understand why changing allocation is a bad thing at first. When analyzing people's behavior you have to take into account factors that may influence this behavior. For instance if your business has a lot of conversions on mondays and if a given variation has more traffic this day, then it will falsely increase its chance to be declared as a winner. Same story on a “weak” day, but then the variation’s performance would be underestimated.

Now you understand why it is a risk to change allocation during an AB test, you can assume that the risk is low if your business activity is constant during the test.

Ok, I can change allocation during a test, but why would I do that ?

There is two mains reasons : 

  • The traffic is so low that there is nearly no hope to reach “statistical significance”.
  • The product that needs to be optimized has a too short life cycle (examples : Flash sales, news article headline, …). In this case a classic AB test would give its result too late, either the product does  not exist anymore or only gets marginal traffic.

So when using dynamic allocation, unlike an AB test, no real decision will be made, instead of a decision (declaring a winner) the test will only change allocation :  the better a conversion performs, the more traffic share it will receive. Doing so does statistical optimisation. We cannot be sure of each specific visitor allocation, but considering the experience as a whole, visitors will be statistically more oriented toward good or best performing variations. In fact dynamic allocation does not declare a winner in the classical meaning, especially because even the worst variation will still receive a little traffic, just enough to ensure that this variation has been tried enough.

All the intelligence in dynamic allocation is in the formulas that choose the best compromise with the limited information it has at each time it updates the allocation scheme.

How does dynamic allocation tests end then ?

You can decide to end it depending on business constraints, for instance for flash sales or news headlines, the business can certainly give guidelines.

You can also decide from the report page. If one variation has a high “chance to win” according to your settings, then you can consider that there is no more usefulness to drive traffic to the other variations.

 

To set that threshold you should understand the fundamental meaning of “significance”. Especially to the fact that the classic AB test suggests using a threshold on statistics to declare a winner. A classic AB test has an original variation sometimes called the reference, this variation already exists and is trusted, where other variations are often mockups that were never tried before. The trust we have in the original comes from the fact that it has historical data that are considered as ok (perfectible but ok). That’s why in a classic AB test, even if B has a little more conversions than A (the reference), we will choose A as a winner. It’s only when the amount of extra conversions is above a given threshold (asserted by a statistical test) that B will be considered a winner. This margin is here to ensure that B is not winning “by chance” against the trusted original.

 

So now if you are using dynamic allocation this means that you have low traffic or at least that the original didn’t go through a strong history of data. So in this context one can use a lower threshold (like 90%).

 

We can even go farther. If all your variations are new, for instance if the flash sales haven’t started yet, or if you are creating several headlines of an article before publishing it, then you do not need to reach statistical significance at all. If you have to make a decision at a given time, then it is totally ok to pick the best variant even if it's only slightly the best.

 

Technical properties : 

  • Dynamic allocation is updated every 10 minutes.
  • It is very rare to have a really stable KPI, so we applied a “slow down” technique in order to adapt the raw statistical technique to reality. So in practice this algorithm will never totally shut down a variation’s traffic. So in practice, if you think that the winner should not chance, then a dynamic allocation test should be stopped at some point.

Conclusion

Dynamic allocation is a technique that can be applied when classic AB Test is not possible because of lack of time, or traffic. This technique is only applicable in a given context (typically with stable KPI). If you have enough traffic, and are not sure that the KPI to optimize is stable then you should prefer the classic AB test.

However, we recommend not using dynamic allocation in the following case:

  • To optimize a meal ordering website. Since allocation changes over time, the primary goal should be stable during the experiment.

⚙️ Configuration

Before switching to dynamic allocation, you need to define at least a primary goal in the Goal step of the test creation flow. The Dynamic allocation algorithm is exclusively based on this primary goal. This means that it detects the best performing variation based on the increase in conversions of the primary goal. For more information, refer to Configuring goals.

🚩Heads up

The primary goal can be any type of goal except Browsing metrics (Revisit rate, Pages per session and Bounce rate).

To activate dynamic allocation, click the Change/switch to Dynamic allocation button at the top of the page. 
Switch_to_dynamic.jpg

A confirmation message is displayed. By confirming, the traffic allocated to each variation will be determined based on the predictive algorithm and on the primary goal you configured for your test. At this point, you won’t be able to manually adjust the percentage of traffic allocated to each variation. However, you are still able to define the percentage of traffic you want to allocate to your test and the percentage of traffic you want to leave untracked.
Back_to_static.jpg

🚩 Heads up

If you didn’t select a primary goal, you won’t be able to switch/change to dynamic allocation. In this case, you will be asked to define at least a primary goal and will be redirected to the goal step.

Once the test has been launched, you can access the reporting anytime, as you would for any campaign with static allocation.
From the test dashboard, tests using dynamic allocation are easily identifiable thanks to the Size_32px__Dynamic_On.png icon.

Once your test has been launched with dynamic allocation, you won't be able to perform the following actions:

  • Change the primary goal
  • Add, remove or duplicate a variation within the test
  • Switch back to static allocation

If you want to perform any of these actions, you need to pause the test and duplicate it.

 

Need additional information?

Submit your request at product.feedback@abtasty.com

Always happy to help! 

ALY-PROF.png

Was this article helpful?

/