Updating Projects with Continuous Optimization

This article explains how you can update a project using Continuous Optimization.

Evolv is designed to allow for an iterative experimentation program that can change over time. This means you can get started with your optimization program right away, instead of waiting to start until you have all of the perfect ideas to test. You can begin the project with the first batch of good ideas, and then update it over time when you get new ideas and your knowledge grows. We call this capability Continuous Optimization.

As participants are entered into a project version, Evolv reports on overall project performance, but also provides insights into which variables and variants are successful or unsuccessful. This tight feedback loop allows for system users to control the optimization strategy - removing unsuccessful variants, and expanding on promising ideas.

Why leverage Continuous Optimization?

• Overcome the burden of designing and implementing many ideas just to get started
• Ability to react to learnings without resetting performance results
• Agility to update due to external factors without losing learnings

What actions does it support?

• Adding, removing, or updating:
• Variables
• Variants
• Targeting Criteria for a variable/group
• Targeting Criteria for a project audience (requires Eng support)

Exceptions

Continuous Optimization does not apply 1) during the initialization phase, 2) when a different audience or optimization metric is selected, or 3) when stopping & re-launching a project.

When to update a project with Continuous Optimization

Continuous Optimization only applies during the Optimization phase of a project.

The initialization of each variant, collecting individual performance data, must be complete before making any changes. This initial data guides our automated decision-making.

Any changes made when a project is in the Initialization phase will result in the start of a new version, resetting performance results.

You can understand how long to wait before Optimization begins by reading the project phase description:

Use Continuous Optimization when adding ideas to your project

The Variants report highlights the ideas and hypotheses that are successful and can inform the next iteration of ideas to add to your project.

Use Continuous Optimization when deactivating variables

When to deactivate a variable:

• When control is declared statistically significant
• When all ideas are substantially below control (dependent on customer risk profile)

Use Continuous Optimization when disabling variants

When to disable a variant:

• When the variant credible interval top end is below zero
• Or earlier, depending on the customer's risk profile

How Continuous Optimization handles updates to variants

Removing a variant

• All combinations that contain the variant will be deactivated
• A clone of these combinations will be created and activated, replacing the removed variant with control

Adding a variant

• The top N (3) combinations will be cloned, adding the new variant to each
• These combinations will then be queued to be brought in after existing combinations have been culled

Updating variants

• The system views a change to variant values as a new variant altogether, on the basis that normally it is a significant change that will affect behavior, rendering the old data invalid.

How to update a project with Continuous Optimization

• Navigate to the Draft stage of your project where you can:
  • Select the Feature Variables section from the left menu to update the variables and variants
  • Update the audience
    • Select a different audience from the drop-down to change the project audience
    • Note: updating the audiences changes the participants being measured and the previously collected data will not be retained in the statistics
  • Update the Optimization Metric
    • Select a different event from the drop-down to change the Optimization Metric
      • Note: updating the Optimization Metric changes the goal of the project and the previously collected data will not be retained in the statistics
      • Update other predicates

Once the project is configured as desired, follow the project workflow to make the changes live