A/B testing experiences contain different creative versions allocated to different traffic to test which version performs better to reach your goal. You can view the A/B testing report by clicking on “A/B Testing Analysis”.
You may have multiple goals to evaluate your testing. Define a key metric on the top-left first and then the whole report will be generated based on the key metric you defined.
Before analyzing a report, it is important to understand Uplift and Probability to Be Best.
Uplift is the difference between the result of a version and the baseline version. The baseline version usually is the control group or the first version when you don’t have a control group.
For example, if your key goal is the purchase, the purchase rate of one version is 5%, and the control group is 4%, the uplift is 25%.
Probability to Be Best
Probability to Be Best is the chance of a version performing better than all other versions in the long term. This is the most actionable metric in the report, which is the key to define the winner of the testing. The probability to Be Best takes sample size into account to make sure the result is reliable based on the Bayesian approach.
A winner will be declared at the top of the report if the following conditions are met:
The estimated goal rate means the goal reached rate you can expect to see over the long run based on current data. We provide 95% probability, 50% probability and the best estimate of the goal rate for each version. As the image shown, the goal rate of this version has 95% probability is between 1% and 3%, and 50% probability is between 1.5% and 2.5%, and our best estimate for this version is 1.9%.
The chart below shows the probability distribution of the goal rate. The blue line represents the selected version, and the black line represents the baseline version. When hovering over the line you can see how the goal rate distributes for each version. At the top of the curve is the goal rate with the highest probability.
How to view the probability distribution chart?
A good way to dig deeper is to break down the result into different properties. This may lead to interesting insights.
Based on your business you can check testing results for different pages that display the testing experience, or for users with different sources, devices, user types, or locations. If you upload user data by Identify Functions, you can also check the testing results for different user properties you uploaded such as user ages, membership level, or industries.
You may find different insights for different breakdown properties. For example, users from campaign X perform better on version A, while users from campaign Y perform better on version B. This may be a good personalization opportunity for you not to send all traffic to a specific version, instead, you can send different versions to different users, which will lead to a higher conversion overall.
If there’s no winner has been declared for a long running time. You can try to:
Was this article helpful?
Thanks for your feedback.
Sorry about that. Do you mind sharing how we can improve?
Write your feedback here...
We have received your feedback and will make sure to improve this article. We will get in touch with you once we do.