This article explains how to understand the relative realized performance value for your project.
Table of Content
What is realized performance?
Realized performance is the difference in conversions seen because we ran the project.
- A project is inclusive of all phases, including Initialisation and Optimization.
If you were an e-commerce manager, you'd use realized performance to report the actual change to the number of converting users while the project is live.
What it is not
It is not how well non-control compared to control. For that, we use cumulative performance.
How realized performance is calculated
The following diagram represents two situations (click the image to enlarge).
- The first is if everyone saw Control as if we didn't run the project.
- There are 100 conversions (10% conversion rate).
- The second is a project with 10% Control.
- Control has 10 conversions (10% conversion rate)
- Non-Control has 180 conversions (20% conversion rate).
- The whole project has 190 conversions (19% conversion rate).
Realized performance is the percentage increase in conversions from running the project.
- The formula for realized performance is:
- "Extra conversions" / "Total conversions at the Control conversion rate"
| Formula | Value | |
| Total conversions | 19% x 1000 | 190 |
| Total conversions at the Control conversion rate | 10% x 1000 | 100 |
| Extra conversions | Total conversions (190) - Total conversions at the Control conversion rate (100) | 90 |
| Realized performance |
Extra conversions (90) / Total conversions at the Control conversion rate (100) |
90% |
90 / 100 = 90%
Multi-phase calculations
To calculate realized performance across phases, you must normalize the results to ensure correct calculations.
You can't add the total conversions across phases because it will be statistically incorrect (E.g., Simpson's Paradox).
To do this, we calculate the extra conversions in each phase, then sum the total extra conversions and divide by the total conversions.
In this example, there are two phases, Initialization and Optimization. Each phase has slightly different conversion rates for Control and Non-Control, and the percentage of traffic to Control is different because the number of concurrent combinations in each phase is different.
Initialization
| Sessions | Conversions | Conversion Rate | |
| Non-Control | 70,000 | 14,000 | 20.00% |
| Control | 5,000 | 500 | 10.00% |
| Total | 75,000 | 14,500 |
To calculate the realized performance for the Initialization phase:
| Total Conversions | 14,500 |
| Total Conversions at Control CR | 7,500 |
| Extra Conversions | 14,500 - 7,500 = 7,000 |
| Realized Performance | 7,000 / 7,500 = 93.33% |
Optimization
| Sessions | Conversions | Conversion Rate | |
| Non-Control | 110,000 | 20,625 | 18.75% |
| Control | 22,000 | 2,090 | 9.50% |
| Total | 132,000 | 22,715 |
To calculate the realized performance for the Optimization phase:
| Total Conversions | 22,715 |
| Total Conversions at Control CR | 12,540 |
| Extra Conversions | 22,715 - 12,540 = 10,175 |
| Realized Performance | 10,175 / 12,540 = 81.14% |
To calculate the realized performance for the whole project across both phases:
| Total Conversions at Control CR | 7,500 + 12,540 = 20,040 |
| Total Extra Conversions | 7,000 + 10,175 = 17,175 |
| Total Realized Performance | 17,175 / 20,040 = 85.70% |
Here's an example from the Evolv AI Manager for a single phase
| Total Conversions | 1,167 |
| Total Conversions at Control CR | 5.66% * 19,495 = 1,103 |
| Extra Conversions | 1,167 - 1,103 = 64 |
| Realized Performance | 64 / 1,103 = 5.78% |