How to create a Multivariate Test

A/B testing is an activity that consists of challenging or testing, elements of a website to find out whether an alternative will increase engagement and conversions. A/B testing is also a great way to identify the next innovation worth investing in, or for determining what is minimally viable. 

AB Tasty offers several types of testing campaigns with your departure hypothesis and what you want to challenge.

To learn more about the differences between each type of test, you can discover our definitive guide to finding the best test and if you don’t know what type of test to choose, our virtual assistant Ally can help you. 

You’ve chosen to create a Multivariate Test

A Multivariate Test consists of modifying several elements on one page (e.g. homepage, basket page, etc.) or a string of equivalent pages that share the same layout (e.g. all of your product pages, all of your result pages etc.), and measuring if one of the new combinations of variation(s) will be more or less performant than the original - for example, changing the shape AND the color of a button, generating several combinations of shapes and colors to test.

Part 1: Main ways to create a new campaign

Just like the other types of campaigns, an Multivariate Test can be created directly from the campaign dashboard, audience manager dashboard, Chrome extension, and/or when duplicating and transforming tests. 

• On the test dashboard, hover over the button "Create", and you’ll discover the following list. Click on "Multivariate Test" to enter the creation flow.

  

• AB Tasty has released a Chrome extension to create a campaign directly from your website page. 

  To download the Chrome extension, please follow this link.
  

  For more information about the AB Tasty Chrome extension, please visit our specific section about the Chrome extension in the visual editor guide. 
  

  This option is useful for loading logged pages.

• If you want to duplicate a previous Multivariate Test and create a new one (and potentially modify some elements of the original test setup), you can use the Duplicate option in the dashboard. 

  
  
  

  • Click on the three dots on the right side of the campaign line you want to duplicate
    
  • Click on the first option, Duplicate
    
  • The duplication modal opens
  • Here, you can:
    
    • Modify the account where you want to create your new campaign.
    • Modify the name of your new campaign (otherwise the name will be the same with (duplicate) added at the end).
      

      For more information about duplication and transformation use cases, please refer to our specific article about this topic.

Part 2: The Main Information page

The Main Information page is the first step toward your new Multivariate Test campaign. 

General use case: pasting the URL you want to test

At the top of the page, the dropdown pre-selects “Multivariate Test”. At this stage, you can still change your mind and select a different type of test.

The fields Name and Sample URL are mandatory and must be filled in to be able to click Save.

The Sample URL is the exact page, or a sample of the pages that share the same layout, that you want to test.

In your campaign, the URL will be used: 

• To load the Editor of your choice (Visual or Code)
• As a pre-filled targeted URL in the targeting step

That this won’t be the final targeted page of your campaign. See the targeting article for more information. 

Don’t use URL parameters in the sample URL field, otherwise, you won't be able to save the step.

Filling in a “Hypothesis” field is a best practice, especially for collaboration across teams. Making the objective of the Multivariate Test clear and easily understood in the hypothesis  (e.g. “Testing if the color of the CTA on product pages increases order size”) will help you and your teams avoid confusion between versions in the Visual Editor.

At this point, you can select one of two ways to create your Multivariate Test variation(s):

• Using the Visual Editor (and WYSIWYG features, widgets, and the possibility to add some JS and CSS code)
  
• Using the Code Editor to fully implement your campaign with code
  

Either way, you’ll be able to come back to the Main Information page to switch the editor, or to change from the Code Editor to the Visual Editor, and vice versa. 

Once your choices and setup are completed, you can click Save & Go to the Next Step.

Once you click Save, your Multivariate Test will be generated (with a unique ID) and will appear in the Tests dashboard. When you come back to the Main Information Page, everything will be editable, except for the type of test (the dropdown menu will no longer be available). At this point, if you need to change the type of campaign, you’ll have to start from scratch or transform your current A/B Test to another type of test directly on the dashboard, using the duplication action.

Specific use case: logged pages

By default, when you load a page that requires authentication (e.g. a client account page) or a conversion funnel page, the page displayed in the editor will be empty or will show an error because it often requires session information (e.g. products to be displayed in the cart page).

To work around this issue, you need to inject the source code of the page within the Main Information step of the creation flow. 

There are two ways to do this:

• Using the Load Editor with embedded source code option manually (as shown)
• Using the Apply HTML option within the AB Tasty Chrome extension (see this article)

The first method with source code consists of pasting the page’s source code directly into AB Tasty. To do this: 

• Go to the URL that you want to load in the editor 
• Right-click on the page, and select Inspect
• In the Elements tab of the console, go to the first line of code: the <html> tag
• Right-click on it, then select Copy → outerHTML. You will get the entire source code of the page with all the scripts and information needed
• Paste this code into the dedicated window within the Main Information step
• Click SaveThe page will load in the editor and you can apply your desired changes.

The URL field needs to remain filled with your URL - this is mandatory to save and go to the next step.

Part 3: The content of your campaign

You have two ways to create content for your campaign in AB Tasty: 

With the Visual Editor:

The Visual Editor enables you to visually create the variations you want to test on your website. For example, a blue CTA instead of the current red one, or a new welcome message on your homepage.

The Visual Editor enables you to manage your Multivariate Test by adding or removing new variations, creating visual modifications, and adding action tracking (click tracking) to follow the performance of the element(s) you’re about to test. 

To learn how to use the Visual Editor, discover options, and dive into our widget libraries, please read the Visual Editor Guide

With the Code Editor:

The Code Editor enables you to declare your code (JavaScript, CSS) without having to load the Visual Editor. The Code Editor is the best solution for developers who want to save time. As the feature doesn’t get a preview to execute the code, you can run your campaign in QA mode using the QA Assistant and refresh the tag each time you modify something in your code before refreshing the page.

To learn more about QA, please refer to the QA portion of this guide.

To learn how to use the Code Editor, please read the Code Editor Guide.

Part 4: Goals

In Multivariate Tests, you can’t add goals with our standard goals setup step - it is not available for this type of test.

Only the transaction goal is available and preselected. This is also why, in the editor, you can't add action trackings or tracking widgets.

For more information about Multivariate Test reports, please refer to its specific section in our Reports guide.

Part 5: Targeting

Targeting is one of the most important moments of the campaign setup. 

You have to declare WHERE your campaign should be visible on your website (on every page, on specific pages, on only one page, etc.), for WHOM (all of your traffic, mobile users only, etc.), and in WHICH conditions (after a certain number of pages, when it’s cold outside, etc.).

Please refer to our complete guide about targeting and learn how to set up segments of visitors, saved pages, and saved triggers to save time on your future setups.

Part 6: Traffic allocation

The traffic allocation step enables you to declare if you want to test all your targeted traffic or just a portion of it. It also allows you to change the iso repartition between variations if you need to.

To learn more about traffic allocation, please read the specific section about Multivariate Tests in this complete guide.

Part 7: Advanced options

The advanced options are not mandatory. 

Third-party Integrations (analytics)

We integrate with several analytics tools, and you can link your Multivariate Test to your preferred analytics tool(s). By doing this, you will be able to analyze your testing cohorts in your dashboards. To learn how to integrate your analytics tool with AB Tasty, please refer to the following guide (especially if you want to set up the API connection at the account level) to avoid having to set it up in each specific campaign. This way, the connection will be fully automatic (for new campaigns moving forward).

If you want to set up an analytics integration in a specific Multivariate Test, you can find your analytics tool in the dropdown and select it.
Some analytics tools, like Google Analytics, require additional settings, which appear in a form panel.

Once launched, your test will start sending data to all the analytics service(s) that you configured here.

Sequential Testing Alerts

We added a Sequential Testing algorithm enabling you to detect if your experiment will not be successful at all based on your Primary KPI results.

Your Primary KPI has to be based on a Conversion/Transaction Rate in order to feed the Sequential Testing alogrithm.

To learn more about our Sequential Testing Alerts, you can follow this link.

Also, only Users and upper roles will receive email alerts if and only if they didn’t disable it in their profile, Notification center section.

Part 8: QA

The QA portion of the campaign is one of the most important steps before launching your campaign into production. 

Please refer to our QA & QA Assistant guide for detailed information on QA.

Part 9: Scheduler

If you want to launch your Multivariate Test at a specific hour on a specific day, and/or if you want to pause it at a specific hour on a specific day, please refer to our guide to scheduling.