Experiments (A/B Testing)

Note: A/B Testing is feature exclusive to Growth Tier customers

Set up experiments to test different variations of your landing pages. This will help you determine which design or content elements are most effective in resonating with your audience. Below are some great examples of experiments to run today!

Core Concepts

An experiment within the Replo A/B testing platform is comprised of the following:

• Name - Generally used for identification
• Experiment URL - By default this is made up of the base URL and the experiment slug, any traffic directed at this URL will be split across all variations. See the section below on how this can be changed.
  • Base URL - By default this is https://reploedge.com/<UUID>
  • Slug - A unique string used to identify an experiment (must be URL encoded)
• Description (Optional) - Generally used by you to stay organized. This is a great place for the goals of the experiment to be documented.
• Variations - Sometimes called “test variants” these are all the different locations the Target URL will direct traffic to. You can have as many variations on an experiment as you’d.
  • Variation Name - This will uniquely identify each variation across the experiment.
  • Variation URL - The actual URL you want to test against other variation URLs. This URL can be any valid URL although it’s typically best for these to be content on your Shopify store.
  • Variation Percentage - Usually but not always split evenly, these allow you to direct more traffic to certain variations if needed. Must add up to 100% across all variations.

How to setup an Experiment

1. Choose Pages to Test

   • Begin with a clear understanding of the pages you want to compare. For example, you might have a landing page version 1 and version 2 with different layouts.

2. In the upper left portion of the Replo editor, find and click into the Experiments button.

   

3. Click New Experiment

   

4. Name the Experiment and set it’s Description value appropriately. These are purely for your organization.

5. Setup a unique experiment slug. It’s best to keep these organized across experiments so you should come up with your own naming convention.

6. Setting Variations:

   • Copy the URLs for the pages you’d like to test into each variation, creating new ones as needed.
   • Optionally you can give each variation a name that makes sense to you
   • Make sure the sum of all of the percentages equal 100%. These are normally an even split across each variation.

7. Click “Save” to create an Experiment in the Draft state

8. Once you’re ready, click “Activate” and your target URL should be live and working in under 60 seconds.

From here you can direct traffic to the Experimental URL and the traffic will be split as defined by the variations configs. Replo splits the traffic to a given experiment URL, say:

https://reploedge.com/<UUID>/ab-test-123

To each variation URL, based on the percentages set, with two extra parameters appended on in the following way:

https://yourstore.com/variation-1-page-url?replo-experiment=ab-test-123&replo-variation=variation-1-name

Specifically:

• The replo-experiment key receives the value of the experimental slug
• The replo-variation key receives the name of the variation as its value

Both of these values are set in the experimental configs as outlined above. If you’d like extra parameters passed along, these can be hardcoded at the end of the Variation URLs (just add them at the end of the URL after a ?)

Send ad traffic or generally link to the Experiment URL and use your existing analytics platforms to understand the experiment’s results.

How do Analytics Work?

Replo does not provide any analytics capabilities within the Replo platform. Instead, we allow your existing analytics platforms to filter the experimental results via the replo-experiment=<experimental_slug> and the replo-variation=<variation_name> parameters added onto every redirect.

This assumes that you have properly installed your analytics platform pixels on your variation URLs. Generally, if these stores are on your Shopify store, this should come out of the box.

As an example, on any report within Google Analytics, you can simply press the “Add Filter +” button:

and build a filter off of any “query string” dimension like so:

Do Variation URLs need to be Replo Pages?

Not technically. These URLs can technically be any valid URLs. You could in theory test Google vs Yahoo. While we’d love you to test Replo pages vs other Replo pages, there are certainly many tests that could be run with Replo content vs non-Replo content (like a non-Replo PDP). See examples of good tests to run below.

Branded Experiment URLs

By default, the experiment URLs will look like:

https://reploedge.com/<uuid>/<experimental_slug>

Which is not a particularly branded experience. This might not be an issue since most consumers should in theory only see this URL for a few milliseconds before their browser loads the URL of one of the variations, but there are options if you want a more branded experiment URL. To configure these, head over to the “Project Settings” view underneath the “Experiment” tab of the left navigation menu.

From here you have two options, the first is simple, and the second more complex.

Project Short Name

The "Project Short Name" within your Project Settings view can be changed from the default random <UUID> to something more branded to your store:

If you change this, your Experiment URLs will take look like the following (as an example):

https://reploedge.com/simple-branded-string/<experimental_slug>

(so the default <uuid> has been replaced with simple-branded-string)

This is a simple way to create a branded experience without changing your DNS settings.

Custom Domain

This is the more complex of the two options as it will require you to point a CNAME record for a simple subdomain of your site at reploedge.com within your domain provider’s DNS settings. To do this:

1. Pick a simple subdomain string, usually something like go or try

2. Add a CNAME record for a subdomain within your DNS settings. The “Name” should be whatever subdomain you choose in Step 1 and the “Target” should be reploedge.com. The exact steps will vary by domain provider but here are a few helpful articles:

   • Shopify Domains Help Center Article
   • GoDaddy Help Center Article
   • Google Domains Help Center Article

3. In the Project Settings view within Replo, add in this full subdomain URL for your store (so go.yourstore.com) in the Custom Domain form and click “Check”. Assuming you’ve done this right, we’ll be able to validate the DNS settings look good (NOTE DNS settings sometimes take a few minutes to propagate).

   

Cookies

Replo will cookie any browser that hits the Experiment URL to ensure that once a sees one variation, that device will always see that variation. This keeps A/B testing results from being cross contaminated and muddled for repeat visitors.

• This cookie is set at the device/browser level
• This cookie is NOT set at the user (individual identity) level

As such, in order to see different variations while testing the Experiment URL you’ll need to use an incognito window or an entirely different browser (Safari vs Chrome).

Experiment Statuses

There are 3* experiment statuses that are important to outline:

Draft - Once an experiment has been created, it will go into a Draft status. Experiment URLs in this status don’t work. From here you can set the experiment to “Active”, “Archive” or “Delete” it.

Active - Once you’ve activated an Experiment from a “Draft” status to “Active” status, the Experiment URL will start to work (sometimes this takes ~30 seconds to go live). The “Activated At” time is also updated to reflect the most recent change from “Draft” to “Active” status.  Once an experiment has run its course (say for 2 weeks), you can mark it "Complete". You can also reset an “Active” experiment to a “Draft” status, “Archive” or “Delete” it.

Completed - Marking an experiment "Complete" will allow you to choose a variation as a winner which will then force all traffic to the Experiment URL to redirect to the winning variation. You can reset a "Completed" experiment to a "Draft" status, “Archive” or “Delete” it.

Archived - This is typically considered the Status for experiments that have run their course. Experiment URLs in this status won’t work and well update the “Archived At” timestamp of the experiment to reflect the latest status change to “Archived”. You can “Unarchive” an experiment which will return it to “Draft”, or “Delete” it.

* Deleted - Generally, you should “Archive” experiments rather than delete them. Only “Delete” if you’re sure you no longer need this experiment or just want to clean up your Experiments view. Deleted experiments can’t be undeleted.

Making changes to Active Experiments

Changing experimental configs mid-flight is technically allowed but not advised. It’s advised that you change any experiments back to a “Draft” state, make any config changes, then “Reactivate” your experiment. Please note the main exception here is you must “Reactivate” your experiments for any Branded Experiment URL changes to take effect.

Example Test Ideas

There is no end to the concepts you can explore with A/B testing. It can truly be overwhelming. In general, the more scientific you’re able to be the more reliable your findings will be. Here are a few ideas to get you started:

• Have a Landing Page for an offer of Product A you’re interested in trying? Setup and run a few hundred dollars of Ad Spend a day at a 50/50 split test between Product A’s product page and your new Landing Page.
• Have a landing page that’s underperforming and you think it’s because the CTA isn’t obvious enough? Copy the landing page and move around sections/buttons as needed (should be as easy as command+C command+V with Replo pages) to add gravity to your CTA.
• Want to test which brand colors work best? Setup any number of landing pages with slightly different color schemes, and run traffic to all of them in one test, measuring your tried and true performance metrics in your favorite analytics platforms.
• Want to increase AOV? Test various versions of offerings and bundles using multiple Add Product To Cart interactions to see how much your customers will purchase from one page.