Skip to main content

Publishing a project

Johnathan Andersen
  • Updated

This article describes what’s required to publish a project and the implications of publishing different changes in different project types. 

Table of Contents

Overview

Publishing is the process of deploying project changes to a specific environment. These changes may include introducing or deactivating variables and variants, updating targeting criteria, or patching broken code in your project.

The publishing process assesses the latest drafted changes to safeguard the integrity of statistical calculations in live projects. It may give you one or more options to proceed depending on the type of project you are running.

In the Manager, this means moving changes from Draft to Live by clicking the “Move to Live” button and confirming your decision to go live.

A record of all published changes can be found on the Activity page of the project’s Live tab. 

Minimum requirements for publishing a project

The minimum requirements for publishing a project are:

How to publish a project

The following instructions describe how to publish a project in the Manager.

  1. Navigate to the project Draft page in the Manager
  2. Select the environment you want to publish to using the menu near the top left of the page
  3. Review the Drafted Changes summary on the Draft Overview page. Use the Advanced view to compare code differences. This is important because the drafted changes can differ between environments.
  4. Click the Move to Live button near the top right of the page
  5. Review the dialog message
  6. If the existing conditions of your changes may impact the live version of the project, then choose the option that’s appropriate for your situation
  7. Enter the desired traffic limit
  8. Click Confirm

The following table describes the implications of different publishing conditions for an A/B Test or Continuous Optimization project.

What to expect when publishing your project for the first time

Continuous Optimization A/B Testing

This action will start a new version of the project.

Publishing Options

Choose which method to determine the size of the project.

After publishing: Initialization

  • Since this is the first time the project is being published, the project will begin a new version with an Initialization phase in which each variant is measured in isolation. 
  • Initialization requires about 400 conversions per variant.
  • New variants added during this time will be initialized after the transition to the Optimization phase.

Later on - Optimization

  • After the system has collected sufficient data on each variant, it will automatically transition to the Optimization phase and use the variant performance data to generate new combinations of variants from multiple variables. 
  • New variants added during the Optimization phase will be initialized first and then added to new combinations. 
  • Initializing new variants is limited to 20% of available traffic to reduce risk to the overall audience.

This action will start a new version of the project.

After publishing

Traffic will be equally distributed to each variant. Regularly check the performance to identify any statistically significant performance.

Recommendation

Create a Continuous Optimization project if you want to add new variables and variants over time.

What to expect when updating your project

Publishing a new variable with variants

Continuous Optimization A/B Testing

Warning – New variables with customized control values can skew performance data. 

Publishing Options

You have a choice to continue running the current version or start a new project version. Starting a new project version will not retain existing performance data.

  • If the control variant has not been modified from its original design, you can continue running the current project version
  • If the control variant has been customized, starting a new version is best practice and statistically sound.

What’s Next - Initializing New Variants

  • New variants added during the Optimization phase will be initialized first before being combined with other variants in new combinations. 
  • New variants will be listed as queued and initialized in random order.
  • All variants in a new variable must be initialized before they can be included in new combinations with other variants.
  • Initializing new variants is limited to 20% of available traffic to reduce risk to the overall audience.
  • [estimated time to initialize all new variants]

Warning – You cannot add new variables and variants to an A/B test

Publishing Options

You will be required to start a new version of the project. Starting a new project version will not retain existing performance data.

Recommendation

  • Create a Continuous Optimization project to add new variables and variants over time.

Publishing a new variant to an existing variable

Continuous Optimization A/B Testing

Publishing Options

You may continue running the current project version without affecting statistical integrity.

What’s Next - Initializing New Variants

  • New variants will be initialized before being added to new combinations 
  • New variants added during the Initialization phase will not be initialized until the Optimization phase has begun.
  • New variants will be listed as queued and initialized in random order.
  • Initializing new variants is limited to 20% of available traffic to reduce risk to the total audience.

Warning – You cannot add new variables and variants to an A/B test

Publishing Options

You will be required to start a new version of the project. Starting a new project version will not retain existing performance data.

Recommendation

  • Create a Continuous Optimization project to add new variables and variants over time.

Updating an existing variant

There are two reasons to update a variant that have different implications and processes.

  1. Addressing a major issue – Major issues are ones in which the performance data collected to date should be ignored after the variant is fixed. For example a variant that breaks the customer experience.
  2. Addressing a minor issue – Minor issues are ones in which the fix or change would not change the variant in a substantial way. For example, adjust the shade of a button color or fixing a typo.

The following describes what to expect in these scenarios:

Addressing a major issue

If you’re addressing a major issue, simply modify the variant in Draft and publish to the desired environment. The system will automatically recognize changes to the variant as a ‘new’ variant and handle it appropriately when publishing. The old version of the variant will be deleted.

Continuous Optimization A/B Testing

Warning

  • All active combinations that include the old version of the variant will be deactivated immediately.

Publishing Options

You may continue running the current project version without affecting statistical integrity.

What’s Next - Initializing New Variants

  • New variants will be initialized before being added to new combinations 
  • New variants added during the Initialization phase will not be initialized until the Optimization phase has begun.
  • New variants will be listed as pending and initialized in random order.
  • Initializing new variants is limited to 20% of available traffic to reduce risk to the total audience.

Warning – You cannot add new variants to an A/B test

Publishing Options

You will be required to start a new version of the project. Starting a new project version will not retain existing performance data.

Recommendation

  • Create a Continuous Optimization project to add new variables and variants over time.

Addressing a minor issue

If the issue is minor, and you don’t need to ignore the existing performance data, then you can use the ‘Restore and Update’ feature to make the change without the system treating it as a new variant.

Continuous Optimization or A/B Testing
Warning

All active combinations with an updated variant will be updated immediately.

Publishing Options

You may continue running the current project version without affecting statistical integrity.

Updating the Optimization Metric

Continuous Optimization or A/B Testing

Warning – You can not change the optimization metric while a project is running because it will impact the statistical calculations to date.

Publishing Options

You must start a new version to measure performance against the new metric. Starting a new project version will not retain existing performance data.

Updating the Project Audience

Continuous Optimization or A/B Testing

Publishing Options

You may continue running the current project version without affecting statistical integrity.

 

Still have questions about what to expect when publishing your project? Submit a support request describing your situation and a member of our support team will reach out to you.

Related articles