Reporting Copilot

The Reporting Copilot is a module aiming at simplifying the analysis process of your campaign data to enable you to make the best decisions. It indicates whether the campaign has a winning variation, what is the winning variation and why.

 

The Reporting Copilot is available only if your campaign: 

  • is an A/B Test or a Multipage test
  • has a transaction goal set as the primary goal
  • has a readiness that has been reached
  • has some traffic on the original version and more than 1000 unique visitors

 

The Reporting Copilot is statistically valuable since it is based on the confidence interval and chance to win both Average order value and Conversion rate of all variations of the reporting. However, you must still consider other goals’ results to make the good decision.

 

The Copilot elements

The Copilot is made of the three following elements:

  • The analysis status
  • The variation highlight
  • The analysis dialog box

Analysis status

This is the result of your campaign and is displayed under the main readiness element for the primary goal. You can see at a glance what is the result of your campaign.

Variation highlight 

According to the campaign result, the winning variation (or the one with the best trend) is highlighted in the goal spreadsheet with specific background colors and captions:

  • When there is a winning variation: the variation is highlighted in green with the mention “winning variation”

  • When no winning variation has been identified: the original version is highlighted with the mention “baseline is winning” in red to show that the campaign is “losing”.

  • When the campaign is neutral: the variation that has the best performance for the moment is highlighted with the mention “trending variation” in yellow.

Analysis dialog box

The campaign analysis gets automatically displayed in a dialog box over the primary goal spreadsheet of the reporting. It can give seven different analysis results (see Use cases section).

 

You can decide not to display the analysis dialog box again by checking the “Don’t show analysis anymore” box (The preference will apply only to this reporting). You can display the dialog box again by clicking on the Report Copilot status button (“Open analysis”).

 

Campaign trends

The reporting can be categorized under three trends:

  • Positive trends
  • Neutral trends
  • Negative trends

Those trends are calculated based on the Average order value and the Transaction rate.

 

Positive trends

Positive trends indicate that we have a winning variation and happens in two situations.

Well performing campaigns

This indicates that the variation has a great optimization potential and can be implemented on production without major risk, as either the average order value or the transaction rate has a good trend while the other has a balanced trend.

 

Over-performing campaigns

This indicates that both the average order value and the transaction rate have good trends.

 

Neutral and opposite trends

Neutral trends indicate that there is no variation recommendation, as either both the average order value and the transaction rate have balanced trends, or they have opposite trends: one performing well while the other is not performing well.

Neutral campaigns due to lack of data

Even when the campaigns have reached the readiness threshold, Copilot currently cannot designate a winning variation. If the campaign has been running for an extended period, further monitoring may be necessary. If the campaign is relatively new, we suggest allowing it to run for a few additional days to gather more data for a better evaluation.

Opposite trends

While a specific variation may be highlighted as trending, we do not recommend implementing it due to the lack of a proper decision-making basis. Despite observing opposite trends in the variations, it's important to note that these fluctuations do not align with the objectives of this particular test. 

Negative trends

Negative trends indicate that the original version (baseline) is performing better than the variations, that is, there are no winning variations identified. 

No winning variation

In such cases, we recommend pausing the campaign and launching a new one with a different approach, typically by testing a new hypothesis.

 

Campaign readiness not reached

When the readiness of your campaign (readiness of your primary goal) has not been reached yet, the Copilot can’t analyze your reporting data.

In this case, you must wait for the three indicators of readiness to be reached. For more information on reporting readiness, refer to Understanding reporting readiness.

When switching the baseline, the Copilot’s logic remains the same: we keep on identifying the potential of the variations compared to the baseline. Keep in mind that the Copilot’s advice can change if the baseline changes.

Was this article helpful?

/