Creating an A/B Test

A/B testing is an activity that consists of challenging or testing, elements of a website to find out whether an alternative will increase engagement and conversions. A/B testing is also a great way to identify the next innovation worth investing in, or for determining what is minimally viable.

AB Tasty offers several types of testing campaigns with your departure hypothesis and what you want to challenge.

To learn more about the differences between each type of test, you can discover our definitive guide to finding the best test and if you don’t know what type of test to choose, our virtual assistant Ally can help you.

You’ve chosen to create an A/B Test
An A/B Test consists of modifying one element on one page (e.g. homepage, basket page, etc.) or a string of equivalent pages that share the same layout (e.g. all of your product pages, all of your results pages, etc.), and measuring if the new variation(s) will be more or less performant than the original.

1 (1).png

Part 1: Main ways to create a new campaign

Just like the other types of campaigns, an A/B Test can be created directly from the campaign dashboard, audience manager dashboard, Chrome extension, and/or when duplicating and transforming tests.

On the test dashboard, hover over the button "Create", and you’ll discover the following list. Click on A/B Test to enter the creation flow.
AB Tasty has released a Chrome extension to create a campaign directly from your website page.

To download the Chrome extension, please follow this link.

For more information about the AB Tasty Chrome extension, please visit our specific section about the Chrome extension in the visual editor guide.

This option is useful for loading logged pages.
If you want to duplicate a previous A/B Test and create a new one (and potentially modify some elements of the original test setup), you can use the Duplicate option in the dashboard.
- Click on the three dots on the right side of the campaign line you want to duplicate
- Click on the first option, Duplicate
- The duplication modal opens
- Here, you can:
  - Modify the account where you want to create your new campaign.
  - Modify the name of your new campaign (otherwise the name will be the same with (duplicate) added at the end).
    
    For more information about duplication and transformation use cases, please refer to our specific article about this topic.

Part 2: The Main Information page

The Main Information page is the first step toward your new A/B Test campaign.

General use case: pasting the URL you want to test

At the top of the page, the dropdown pre-selects “A/B Test”. At this stage, you can still change your mind and select a different type of test.

The fields Name and Sample URL are mandatory and must be filled in to be able to click Save.

The Sample URL is the exact page, or a sample of the pages that share the same layout, that you want to test.

In your campaign, the URL will be used:

To load the Editor of your choice (Visual or Code)
As a pre-filled targeted URL in the targeting step

That this won’t be the final targeted page of your campaign. See the targeting article for more information.

Don’t use URL parameters in the sample URL field, otherwise, you won't be able to save the step.

⭐️ Tips: Filling in a “Hypothesis” field is a best practice, especially for collaboration across teams. Making the objective of the A/B Test clear and easily understood in the hypothesis (e.g. “Testing if the color of the CTA on product pages increases order size”) will help you and your teams avoid confusion between versions in the Visual Editor.

At this point, you can select one of two ways to create your A/B Test variation(s):

Using the Visual Editor (and WYSIWYG features, widgets, and the possibility to add some JS and CSS code)
Using the Code Editor to fully implement your campaign with code

Either way, you’ll be able to come back to the Main Information page to switch the editor, or to change from the Code Editor to the Visual Editor, and vice versa.

Once your choices and setup are completed, you can click Save & Go to the Next Step.

Once you click Save, your A/B Test will be generated (with a unique ID) and will appear in the Tests dashboard. When you come back to the Main Information Page, everything will be editable, except for the type of test (the dropdown menu will no longer be available). At this point, if you need to change the type of campaign, you’ll have to start from scratch or transform your current A/B Test to another type of test directly on the dashboard, using the duplication action.

Specific use case: logged pages

By default, when you load a page that requires authentication (e.g. a client account page) or a conversion funnel page, the page displayed in the editor will be empty or will show an error because it often requires session information (e.g. products to be displayed in the cart page).

To work around this issue, you need to inject the source code of the page within the Main Information step of the creation flow.

There are two ways to do this:

Using the Load Editor with embedded source code option manually (as shown)
Using the Apply HTML option within the AB Tasty Chrome extension (see this article)

The first method with source code consists of pasting the page’s source code directly into AB Tasty. To do this:

Go to the URL that you want to load in the editor
Right-click on the page, and select Inspect
In the Elements tab of the console, go to the first line of code: the <html> tag
Right-click on it, then select Copy → outerHTML. You will get the entire source code of the page with all the scripts and information needed
Paste this code into the dedicated window within the Main Information step
Click SaveThe page will load in the editor and you can apply your desired changes.

The URL field needs to remain filled with your URL - this is mandatory to save and go to the next step

Part 3: The content of your campaign

You have two ways to create content for your campaign in AB Tasty:

With the Visual Editor:

The Visual Editor enables you to visually create the variations you want to test on your website. For example, a blue CTA instead of the current red one, or a new welcome message on your homepage.

The Visual Editor enables you to manage your A/B Test by adding or removing new variations, creating visual modifications, and adding action tracking (click tracking) to follow the performance of the element(s) you’re about to test.

To learn how to use the Visual Editor, discover options, and dive into our widget libraries, please read the Visual Editor Guide

With the Code Editor:

The Code Editor enables you to declare your code (JavaScript, CSS) without having to load the Visual Editor. The Code Editor is the best solution for developers who want to save time. As the feature doesn’t get a preview to execute the code, you can run your campaign in QA mode using the QA Assistant and refresh the tag each time you modify something in your code before refreshing the page.

To learn more about QA, please refer to the QA portion of this guide.

To learn how to use the Code Editor, please read the Code Editor Guide.

Part 4: Goals

Once you’ve created your variation(s), you need to pre-configure your future reporting to easily identify the metric(s) that you want to track as your primary goal and (optional) secondary goals. The primary goal is the goal that has the potential to be most impacted by your modifications in the A/B Test (e.g. if the A/B Test consists of modifying the color of the CTA, the click rate on the CTA would be your primary goal).

To learn more about the different types of metrics and how to set them up, please refer to the following guide.

The metrics are split into different sections:

Action Tracking: all the trackers you’ve set up in the Visual Editor (not retroactive)
Page Tracking: the pageviews you’ve recorded in your account and an option to create new ones
Custom Tracking: the action tracking and other trackings you’ve set up by code in the editor or the global code of the account
Browsing metrics: includes bounce rate, pages per session, and revisit rate (retroactive)
Transaction metrics: includes the transaction indicators you’ve set up on your account, which you can access only by installing our transaction tag on your website.

To learn about how browsing metrics are calculated, check out this resource.

Selecting your primary and secondary goals

During this step, you will need to select a primary goal, which will serve as the primary point of reference for your campaign. You can also select several secondary goals to track in the campaign reporting. Only the goals that you select will appear in your campaign report.
You can add or remove secondary goals down the line (in this case, they will be retroactive), but the primary goal cannot be changed once it has been saved. However, the goals set in the editor (Click Tracking, Action Tracking) are not retroactive. You must configure them before launching your test.

Click on the category related to the goal you want to set: A list of indicators appears on the right-hand side.
Click the + icon to the right of the indicator you want to select: The goal is displayed in the right-hand pane as goal no. 1 in the Primary Goal category.

Once your goals have been saved and your campaign has started, you can no longer modify or remove the primary goal of your campaign.

Once your campaign has started, you can add secondary goals, but only pageviews will be retroactive in the reports. All the other goals will be calculated from the date you added these metrics as goals. You can add up to 49 secondary goals.

Save before quitting the page. Your goal setup will serve as the basis of your future reporting when analyzing the results of your test. Each chosen metric will appear as a goal in the reporting.

Hiding transaction indicators

When you implement a transaction tag, the transaction affiliation (ta) name you assigned to your transaction tag automatically appears as a transaction indicator in the Transaction section of the Goals Setup step of a campaign.

For more information on how to configure a transaction tag, please refer to installing the transaction tag.

The more transaction tags you implement, the longer the transaction indicators list will be. These transaction indicators can then be chosen as primary or secondary goals for your campaign.

The transaction indicators’ Hide option enables you to clear and lighten the transaction indicators list by hiding out-of-date transaction indicators.

This way, it becomes easier and more convenient to find the transaction indicator(s) you want to set as a primary or secondary goal(s) for your campaign, thereby removing the obsolete one(s).

Before deleting transaction indicators, and if you have made any modifications in the step (such as adding or removing a goal), click Save to make sure the goals are up-to-date.

To hide transaction indicators from the Transaction list, apply the following steps:

Click Hide transaction indicators
From the pop-in, hover over the indicator you want to hide from the list and click the trash icon
Repeat this action for the other indicators you want to delete
Click the Hide button to confirm

⚡️ Heads up: You can’t hide an indicator that is used in one or several other campaigns, whether they are live or paused. This is disabled.

💡 Good to know: The hidden transaction indicators will be hidden from the list for all campaigns, but not deleted permanently. For legacy reasons, to delete one, you must do so directly in your website code.

Retrieving the hidden transaction indicators

You can retrieve the list of all the transaction indicators you have hidden in your account settings from Account Management → Transaction Indicators Management.

To reactivate one, hover over it and click Activate. The indicator will disappear from this page and you’ll be able to see it in the transaction indicators list in the Goals Setup step and use it as a primary or secondary goal for your campaigns.

Part 5: Targeting

Targeting is one of the most important moments of the campaign setup.

You have to declare WHERE your campaign should be visible on your website (on every page, on specific pages, on only one page, etc.), for WHOM (all of your traffic, mobile users only, etc.), and in WHICH conditions (after a certain number of pages, when it’s cold outside, etc.).

Please refer to our complete guide about targeting and learn how to set up segments of visitors, saved pages, and saved triggers to save time on your future setups.

Part 6: Traffic allocation

The traffic allocation step enables you to declare if you want to test all your targeted traffic or just a portion of it. It also allows you to change the iso repartition between variations if you need to.

To learn more about traffic allocation, please read the specific section about A/B Tests in this complete guide .

Part 7: Advanced options

The advanced options are not mandatory.

Third-party Integrations (analytics)

We integrate with several analytics tools, and you can link your A/B Test to your preferred analytics tool(s). By doing this, you will be able to analyze your testing cohorts in your dashboards. To learn how to integrate your analytics tool with AB Tasty, please refer to the following guide (especially if you want to set up the API connection at the account level) to avoid having to set it up in each specific campaign. This way, the connection will be fully automatic (for new campaigns moving forward).

If you want to set up an analytics integration in a specific A/B Test, you can find your analytics tool in the dropdown and select it.
Some analytics tools, like Google Analytics, require additional settings, which appear in a form panel.

Once launched, your test will start sending data to all the analytics service(s) that you configured here.

Sequential Testing Alerts

Good to know 💡

That feature is currently available in Early Adoption. Please contact your CSM to enroll into the Early Adopter program.

We added a Sequential Testing algorithm enabling you to detect if your experiment will not be successful at all.

To learn more about our Sequential Testing Alerts, you can follow this link.

There’s 2 option to configure in your Experiment:

The type of automation you want to benefit from
The threshold you want to experiment

The type of automation

You can either receive an email if your campaign is failing, a kind of “Warning, you’re experiment is going right into the wall” or directly pause the campaign if it’s failing.

The sensitivity threshold

You have 3 different level of sensitivity here:

High - It’s the safest solution. You’ll be more likely to receive alerts but you’re sure that losing experiments will be detected sooner.
Low - It’s the more advanced setup. You’ll receive less alerts, but your losing experiments will take longer to be detected.
Balanced - It’s the half-way between the 2 previous detection levels.

You can learn more about detection level on the Sequential Testing Alert article.

Good to know 💡

You can only configure this setup if you selected a Primary Goal at the previous step!

Also, only Users and upper roles will receive email alerts if and only if they didn’t disable it in their profile, Notification center section.

Part 8: QA

The QA portion of the campaign is one of the most important steps before launching your campaign into production.

Please refer to our QA & QA Assistant guide for detailed information on QA.

Part 9: Scheduler

If you want to launch your A/B Test at a specific hour on a specific day, and/or if you want to pause it at a specific hour on a specific day, please refer to our guide to scheduling.