The Reporting Copilot is a module aiming at simplifying the analysis process of your campaign data to enable you to make the best decisions. It indicates whether the campaign has a winning variation, what is the winning variation and why.
The Reporting Copilot is available only if your campaign:
- is an A/B Test or a Multipage test
- has a transaction goal set as the primary goal
- has a readiness that has been reached
- has some traffic on the original version and more than 1000 unique visitors
The Reporting Copilot is statistically valuable since it is based on the confidence interval and chance to win both Average order value and Conversion rate of all variations of the reporting. However, you must still consider other goals’ results to make the good decision. For information on the reporting, refer to Campaign reporting.
The Copilot elements
The Copilot is made of the three following elements:
- The analysis status
- The variation highlight
- The analysis dialog box
Analysis status
This is the result of your campaign and is displayed under the main readiness element for the primary goal. You can see at a glance what is the result of your campaign.
Variation highlight
According to the campaign result, the winning variation (or the one with the best trend) is highlighted in the goal spreadsheet with specific background colors and captions:
- When there is a winning variation: the variation is highlighted in green with the mention “winning variation”
- When no winning variation has been identified: the original version is highlighted with the mention “baseline is winning” in red to show that the campaign is “losing”.
- When the campaign is neutral: the variation that has the best performance for the moment is highlighted with the mention “trending variation” in yellow.
Analysis dialog box
The campaign analysis gets automatically displayed in a dialog box over the primary goal spreadsheet of the reporting. It can give seven different analysis results (see Use cases section).
You can decide not to display the analysis dialog box again by checking the “Don’t show analysis anymore” box (The preference will apply only to this reporting). You can display the dialog box again by clicking on the Report Copilot status button (“Open analysis”).
Campaign trends
The reporting can be categorized under three trends:
- Positive trends
- Neutral trends
- Negative trends
Those trends are calculated based on the Average order value and the Transaction rate.
Positive trends
Positive trends indicate that we have a winning variation and happens in two situations.
Well performing campaigns
This indicates that the variation has a great optimization potential and can be implemented on production without major risk, as either the average order value or the transaction rate has a good trend while the other has a balanced trend.
Over-performing campaigns
This indicates that both the average order value and the transaction rate have good trends.
Neutral and opposite trends
Neutral trends indicate that there is no variation recommendation, as either both the average order value and the transaction rate have balanced trends, or they have opposite trends: one performing well while the other is not performing well.
Neutral campaigns due to lack of data
Even when the campaigns have reached the readiness threshold, Copilot currently cannot designate a winning variation. If the campaign has been running for an extended period, further monitoring may be necessary. If the campaign is relatively new, we suggest allowing it to run for a few additional days to gather more data for a better evaluation.
If the readiness has been reached more than 30 days ago, the reporting copilot won't be able to display the "needs more data" status, and you will see either a positive (winning variation), negative (baseline winning) or neutral (opposite trends) status.
Opposite trends
While a specific variation may be highlighted as trending, we do not recommend implementing it due to the lack of a proper decision-making basis. Despite observing opposite trends in the variations, it's important to note that these fluctuations do not align with the objectives of this particular test.
Negative trends
Negative trends indicate that the original version (baseline) is performing better than the variations, that is, there are no winning variations identified.
No winning variation
In such cases, we recommend pausing the campaign and launching a new one with a different approach, typically by testing a new hypothesis.
Campaign readiness not reached
When the readiness of your campaign (readiness of your primary goal) has not been reached yet, the Copilot can’t analyze your reporting data.
In this case, you must wait for the three indicators of readiness to be reached. For more information on reporting readiness, refer to Understanding reporting readiness.
When switching the baseline, the Copilot’s logic remains the same: we keep on identifying the potential of the variations compared to the baseline. Keep in mind that the Copilot’s advice can change if the baseline changes.