Using the Mutually Exclusive Experiments feature

💡 Good to know

This feature is currently available in Early Adoption. Please contact your CSM to enroll into the Early Adopter program.

📖 Definition

Mutually Exclusive Experiment (M2E) is a feature enabling you to launch several tests at the same time, on the same targeting, with the same primary goal without being afraid of any UX conflict or data biases.

To learn more about benefits and traps of M2E, you can read this article [INSERT DATA SCIENCE ARTICLE link]. Mutual Exclusive Experiments

 

⚙️ Configuration

Creating an Exclusion group

Everything starts in the test dashboard.

When clicking on the 3 dots of a campaign, you’ll see 2 options. Either “Add to Exclusion group” or “Manage Exclusion group” depending if your test is already part of a group or not.

To exclude a test campaign from another, select the option “Add to exclusion group”. When done, a modal appears.

In that modal, you’ll be able to select between the creation of a new group starting with this test OR to add this test to an already existing group.

 

You can name your group to better keep track of the test you’re excluding (ie: “Home page”, “Add to cart improvement”, etc.).

Once done, you can select the different tests you want to make mutually exclusive.

Opening the dropdown will give the list of all tests of your account (limiting it to 12 at the first render). You can also search for specific tests should you know the name.

It might be possible that you can’t select a test in the dropdown. There are several reason for that:

  1. The test is already part of a group
  2. The test is already live

Tests can only be part of one group at a time and a test which has already received some traffic can’t be part of an exclusion group.

Heads up ⚡️

Adding a test to an exclusion group decreases its global traffic allocation, meaning that if you would be able to change the allocation in the middle of a test, data would be biased.

 

To see only the test you can add to the current exclusion group, you can click on the “Show available tests only” option.

Then, clicking on Done will add that test to the exclusion group and split the global allocation per campaign.

You can now Save that group and continue your activities!

 

Heads up ⚡️

The more tests you’ll add to a group the more your traffic will be split between campaigns!
Meaning that if you were waiting 14 days to have significant results before, you might have to wait 28 days now with 2 campaigns in a group.
Please, use that feature wisely and ask your CSM if you have any doubt.

Adding to an existing Exclusion group

If your Exclusion group already exists and isn’t locked, you can add your test to it by clicking on the option “Add to existing group”.

You’ll be able to select available groups (groups which aren’t locked yet).

You can still see available groups only by ticking the box “Show available groups only”.

Selecting a group and Saving will directly add your test to the group and you’ll be able to play with its configuration.

 

Group management

In the test dashboard, you can see a tab “Exclusion group” where you’ll have an overview and be able to manage all of them.

The number between parentheses indicates the number of exclusion groups created.

By clicking on the manage button, you’ll be able to view the group configuration or play with it depending on the group status.

 

Unlocked group

An unlocked group is an exclusion group in which tests have never been pushed live. As soon as a test in a group is pushed live, the group is locked and its configuration can’t be edited anymore.

So clicking on the Manage button on an unlocked group will enable you to add or remove tests from the group.

 

When you activate the QA of a test (meaning that you toggle ON the IP Address and/or cookie parameter), your test becomes Live in QA, which means that you can’t add it to any exclusion group. When you are done with the QA (meaning that you toggle OFF the IP Address and/or cookie parameter), a modal displays to ask if you want to clear data collected during the QA.
> If you clear those data and pause your test, you will be able to add your test in an exclusion group. Indeed, for AB Tasty, cleaning the data means that you haven’t collected any data and that the test has never been pushed live.
> If you keep those data (you don’t clear them), you won’t be able to add it in any exclusion group. Since data has already been collected, the test is considered Live “for real”;
You can also clear your data from the reporting of your test:

For this reason, we recommend doing the QA of your test before adding it to an exclusion group, and cleaning data collected during the QA phase.

 

Locked group

A locked group is a group in which a test has been pushed live. As soon as the group has been locked, its configuration can’t be changed. 

Clicking on the manage button will enable you to view the current group configuration.

You’ll still be able to delete the group even if all tests are paused.

 

Deleting a group

You can delete a group at any time as soon as all the tests inside are paused.

If the group was not locked, there would be no impact on campaign configuration.

If the group was locked, you’ll not be able to relaunch campaigns which were already launched, to avoid any allocation issue.

In case you want to do so, the platform will invite you to duplicate the campaign.

 

 

Reporting indication

In the reporting, you’ll see a notification under the number of visitors on the Activity Headers to remind you that your test was part of an exclusion group and the global traffic of your test isn’t the same as usual.

 

💡 Use cases

#1

As a User I need to launch several tests at the same time on the same page. Those campaigns shouldn’t enter into UX conflict as they are acting on different elements but my primary KPI is the same for all campaigns.

In order to avoid any data biases I’d prefer making them mutually exclusive and do a proper analysis at the end.

 

#2

As a User I need to test different parts of my page and I’m afraid of UX conflict between those parts. Thus, to avoid any bad user experience I would better use the Mutually Exclusive Experiments feature and be more confident during the QA.



FAQ

What type of campaign can I make mutually exclusive?

You can add to the exclusion group experiments campaigns (test campaigns) such as A/B Test and Multipage Tests.

How are cookies affected? 

When the user lands on your website, a random test among your exclusion group is assigned to him. As soon as his cookies aren’t erased, he will still be assigned to the same test.

What happens when my visitor lands on my website?

When your visitor lands on your website, the tag randomly assigns him to one of the tests of the exclusion group. Note that the visitor must belong to the targeting rules to see the test.

What about QA?

We advise you to perform your QA before pushing a campaign to a group. When you're QA has ended, Pause your test and Clear your data in order to add the campaign into a group.

Launching QA Assistant when a campaign is in a group would lock the group.

Was this article helpful?

/