Analyzing A/B test campaigns can be challenging. The impact of the tested variation on campaign goals may be non-significant, and finding an audience that shows a clear preference for a variation involves uncertainty regarding both the duration of the test and its success. As a result, the question of whether certain users prefer a given variation often remains unanswered.
EmotionsAI provides a ready-to-use segmentation technology designed to identify groups of visitors with very different preferences regarding their “ideal online experience.” This is why EmotionsAI is so effective in answering questions about understanding A/B test results.
To access these insights, please ensure EmotionsAI insights have been activated on your account (see dedicated guide How to activate EmotionsAI Features), and apply the EmotionsAI template on your campaign reporting (see this how-to guide).
EmotionsAI will bring you value in various ways depending on the type of variation you are testing, which are detailed in the next 2 sections.
Getting more value when testing lightweight variations (low-code and low-impact on page template)
When analyzing results from such variations, the main opportunity that EmotionsAI will highlight is personalization. Variations that are lightweight in code and have minimal impact on the template, are ideal candidates for long-term use in AB Tasty Web Experimentation & Personalization without significantly impacting web performance (thanks to lightweight code) or requiring extensive template maintenance (easy maintenance). The following kinds of variation are typical examples of good candidates: banners, wording changes (especially on CTA button), hiding elements, or adding new elements on empty zones, etc. Don’t hesitate to explore use cases in our online resources to find examples and inspiration.
Even if a variation has a neutral or negative impact on campaign goals, EmotionsAI may reveal specific segments that were more engaged with the tested variation and could benefit from a personalized experience. These detected opportunities are immediately visible in the primary goal results after the EmotionsAI template is applied. If the goal is transactional, you will also see the revenue uplift potential of such personalizations.
On the other hand, if a variation is winning on the campaign, you can select it as the new baseline in order to identify if some EmotionsAI segments prefer this variation over the average preference of the audience. This kind of detection is at the very core of personalization–identifying visitor groups who prefer a different experience than the general audience. After selecting the winning variation as the baseline, you simply have to replicate the previous process and look at the highlight view or detailed view of the EmotionsAI segment template to detect “green” segments.
A last “in-between” scenario is when a variation is almost statistically significant in a positive way, and you hesitate whether to put it in production. In this case, you can use the EmotionsAI template to discover if any segments had a statistically significant negative impact on the campaign’s goals. If such segments exist, you can set up a personalization that will display the variation to everyone except the segments who reacted negatively.
Last but not least, please keep in mind that EmotionsAI segments can be exported to your 3rd party tools, including any back-end personalization tool you might be using. This will allow you to leverage the detection and activation of personalization opportunities with a lower constraint on code weight.
Getting more value when testing code-heavy or template-impacting variations
As explained in the previous section, variations that are either code-heavy or that have a strong impact on the template are not ideal candidates for front-end personalization in the long term. However, EmotionsAI can still provide significant value in these cases by saving time on analysis, decision-making, and potential iterations.
If the tested variation is a winner, EmotionsAI will help you determine whether the success is across the entire audience or specific segments. In the latter case (which is more common), understanding which segments had positive reactions will give you insights into what works well on these segments. This strategic information can be reused elsewhere else in the journey if these segments under-perform at other stages. Now, you’ll have a better understanding of how to improve their experience and save time crafting targeted variations for them.
On the other hand, if the impact of tested variations on campaign goals is negative (whether statistically significant or not), EmotionsAI can help you identify the segments that drove the poor results. This will save you time in understanding the campaign results and also help in iterating a new version of your variation, by already having the right guidelines in mind. For instance, if you observe a strong negative reaction from your visitors in the “Understanding” segment, it suggests that your next experiment will perform better if you add more information or better structure existing information.