This article explains why the system 'contaminates' a session and how to view them in a project or combination.
Table of Contents
What is a 'contamination'?
A contamination event occurs when something unusual happens during an experiment session.
When a contamination event occurs, the user becomes 'contaminated' and is excluded from the performance calculations and overall results seen in the Manager.
Contaminations are monitored daily to ensure experiments are working as expected.
The three primary contamination types are:
- Timeout-Exceeded - A timeout indicates the Evolv snippet could not load and apply changes within a specified window of time. By default, the threshold is 60 seconds. This contamination type is the most common and is often the result of a mobile user with a poor internet connection.
- Error-Thrown - This indicates a JavaScript error thrown by the experiment code. The error is often due to accessing or manipulating an invalid selector. Whenever we come across this type of contamination, we immediately look into the cause and resolve it quickly.
- Project-Paused - When an experiment is paused, any user assigned a non-control combination will begin to see Control. Since the user would technically see two different experiences, we exclude those users from the results.
Impact of Contaminations on Relative Performance
It comes down to whether the contaminations bias your active combinations in a way that benefits one variant or combination over another.
If contaminations are distributed evenly across the candidates it will just cost you traffic.
You have 2 ways contaminations will commonly cause issues
- is that you have a Combination that is erroring out, but Control isn't, leading to an imbalance in traffic between Control and the Combination which will open up the relative comparison to errors
- The other is that the users being contaminated have a different behavior from those that are not being contaminated. For instance, erroring out on IE users who have a lower probability of converting
For example: Imagine users that went to an empty cart page were included into the project, but some variants were resulting in an error that would exclude those people. This resulted in people who hadn't made a buying decision yet get included for most combinations, but not the ones with specific variants, which made them appear to perform exceptionally well.
Steps to check for contaminations in the Manager
All contaminations are recorded in the Manager. You can see the number and type of contaminations for a project or individual combination.
- Open the Evolv AI Manager and go to the LIVE tab for a project.
- On the Overview page, scroll down until you see the table of Total Unique Visitors.
- Below the table, you'll see the Contamination rate for the whole project. Click the Show details link to view the Daily Number of Contaminations.
- Click on a day in the graph to see that day's contamination events.
- To view contaminations for a particular combination, go to the Combinations page and click on a combination.
The Daily Contamination Rate is available when viewing a single day of contaminations. The graph of daily contaminations helps you understand when the contaminations started and stopped.