This article describes how to understand the performance of variants.
Evolv AI evaluates the performance of individual variants within a project. This provides a better understanding of which specific ideas and hypotheses work well or poorly. For example, when you are getting ready to implement a winning combination, you can investigate the performance of each variant in that combination and then decide to leave out specific variants that are not the ones driving the highest performance.
Variants Graph
The variants graph shows an individual variant's relative change from control and the probability the system will show it to a new visitor. Generally, variants with a negative change are less likely to be seen. Variants that perform well will have a higher chance of being shown.
Statistically significant variants are displayed with a solid green indicator.
Performance of Variants
The Performance of Variants table organizes variants by variable and shows stats for individual variants.
Sessions/Visitors
A count of all sessions or visitors who have ever seen the variant.
Probability to be seen
Each variant can exist in multiple combinations that are eligible to be shown to visitors. Each variant combination has an associated probability of being allocated to a user based on its performance. These probabilities are aggregated to generate the probability that any new visitor will see each variant.
The sparkline shows how this value has changed over time.
Expected performance
The credible interval of the variant at a 95% confidence level.
Statistical significance indicator
The indicator appears solid green when the variant has achieved statistical significance, which means that the system has concluded that the difference in conversion rates between a given variant and the control is not due to random chance, with 95% certainty.
Frequently asked questions
My variant performed well initially and is part of combinations that perform well, but its performance is no longer great. Why?
Each variant is initialized independently to establish a performance baseline. Over time, that performance changes as the variant appears in more combinations.
If those combinations perform well and the variant performance worsens, the positive combination performance can be attributed to other variants in those combinations.