This feature is in early beta testing and is only being tested with some select merchants. If you are interested in joining the beta, please send us an email.
A/B Testing allows you to test and compare up to 5 different variations on a single merchandising campaign, each of which you can make a change using any Visual Merchandising tools: Boost rules, banners, pinned products, removed products, and custom facet order.
Setting up an experiment
An experiment is an individual test run on a campaign over a specific amount of time.
To begin setting up an experiment, go to edit a campaign, open up the A/B Test toolbox, and select "New Experiment".
The basic options for an experiment are shown. You'll need to select an End Date.
Under "Set up your variations", select the variation you'd like to set up. Then, exit the toolbox and make all of the changes you'd like. The variation you're currently changing is marked in blue to the left.
Optionally, you can also choose which variation to use as a default after the experiment ends, but before you've selected a formal winner.
Near the top of the A/B toolbox, you'll see the text: "Upon completion, Variation A will be used until a winner is declared by you (Change)". Click on "Change" to modify which variation you'd like to default to.
You can add up to 5 different variations in a single experiment by clicking "Add another variation".
The more variations you add, the longer you'll need to run your experiment to get accurate data sample sizes.
You can also set up what percentage of shoppers are split into each variation by dragging the sliders under "How many customers should be placed in each variation?"
To begin your experiment, simply save the campaign as you normally would. A popup dialog will ask to confirm the beginning of the experiment. You will not be able to make any merchandising changes until the experiment is complete.
Running an experiment
While the experiment is running, no merchandising tools will be editable.
However, you may still edit the campaign's title, schedule, experiment end date, experiment default winner, and end the experiment manually. (See below)
When viewing your list of campaigns, a blue icon will indicate an experiment is currently running.
A/B Test toolbox while an experiment is running
While an experiment is running, you can still use the A/B Test toolbox to change the end date, and the default ending variation.
Ending the experiment manually
To end an experiment immediately, you can click on "End Experiment Now". Here you will need to commit to a winner, which will be used moving forward.
Once a campaign has reached its end date, the A/B Test toolbox will prompt you to select the winner.
A red icon on the list of campaigns will also indicate that the experiment's end date has been reached, and a winner needs to be declared.
In either case, you'll need to save your campaign once a winner has been declared to complete the experiment.
Trying it out
You can easily simulate different shoppers on your live site by clearing your cookies for the site. We suggest a tool such as Cookie Remover for Chrome. Once you've cleared your cookies, simply refresh the page and you will be re-sorted into a group for the experiment. There is a chance that you'll be sorted into the same variation group again, so you may need to do so multiple times.
How do I view reporting for an experiment?
To track the performance right away, you can do so by setting up Google Analytics reports. Reporting within the management console is coming soon.
A/B Testing: The name of this feature.
A/B Test: A common synonym for "Experiment" or "A/B Merchandising"
Experiment: An individual test run on a campaign over a specific amount of time. Contains 2-5 variations in a single experiment.
Variation: An individual set of elements. Shoppers are divided randomly and shown only one variation.
Element: Part of the page's behavior that a merchandiser can change. There are 5 elements: banners, boosts, pinned products, removed products, and custom facet order.
Toolbox: The expandable panels available on the right hand side of the Visual Merchandising Edit Campaign interface. Each toolbox may contain multiple tabs to view/manage other content.
Context: The conditions which must be met in order for the campaign to take affect.