Multivariate testing: what it is & how to conduct it

A/B/n testing is one of the marketing research methods used to compare the effectiveness of several content variations in an email newsletter and other campaigns. It allows you to check individual elements or several different templates. The purpose of testing is to find out which variation works better and brings in more conversions.

The elements of an email template that can be tested with A/B/n test are:

• the amount of text in emails;
• text, design, size or placement of conversion buttons;
• subject or preheader of an email;
• push notification title or icon;
• prices of goods;
• product description and titles of the description;
• illustrations for products.

Why A/B/n testing is necessary

1. It reduces the risks of monetary losses. The easier it is for a customer to find the desired product and place the order, the more likely they are to buy from you. Invisible buttons and complex navigation lead to the loss of customers and potential income.

2. It increases conversion. With the help of this test, you will find the most clickable variations, and your customers will find the desired products and purchase them faster.

3. It helps to test different ideas. Team members can have disagreements during the development of a new design. Whose idea is better? Split testing will show. It will test hypotheses and resolve the conflict peacefully.

4. It saves resources. The test will show which variations are more effective and save your time. It is faster than arguing with each other or testing one version at a time.

5. It provides you with ideas for future campaigns. User interaction with different design options is a source for new, sometimes unexpected, insights.

How to conduct A/B/n testing

In Altcraft Platform you can conduct both simple split tests (A/B) and more complex A/B/n testing. Let's conduct a simple A/B test, i.e. compare two variations of one message, to see how to work with MVT campaigns on the platform.

Step 1. Identify the problem and formulate a hypothesis

Find the weak points of your marketing campaigns. Analytical tools will help you do it. For example, you can use the Altcraft Platform channel report, which contains all main metrics of newsletters in different channels. Let's say that you noticed that the welcome email newsletter has high Open Rate, but low CTR. It means that the subject and preheader are good, and the problem is most likely with the content: irrelevant text, lack of visual content or ineffective call to action.

Another useful tool is click map. It visualizes your email and shows the number of unique clicks of subscribers near each link. This map will help you determine the most successful placement and design of links.

Step 2. Create variations for the test

Now you need to create a separate template for each message variation. You can test up to 8 templates at the same time, but remember that the more variations of the message, the larger your audience should be for the test. The results will be relevant only if there is enough traffic for each variation. Also, to get reliable results it is recommended you test only one element of the template.

Go to «Templates». Click «Create» at the top of the page and select the necessary sending channel: Email, SMS or Push. Make up templates and save them.

For example, we’ve created two email templates for the promotional newsletter «Women's Day Collection». Now we need to test which image is more effective.

Step 3. Create a campaign

Open «Campaigns» → «Multivariate testing». Click on «Create» at the top of the page. Then set the basic settings: the name of a campaign for in-house use, brief description, tags for quick search and access groups. Activate the toggle switch in the upper right corner — «Active». Your newsletter will be sent out as soon as the date specified at the schedule comes.

General MVT campaign settings

In the section below, select the sending channel and click «Next»:

Sending channel

Step 4. Determine the sample size

At this stage, you need to find out how many subscribers should participate in the test. If there are not enough people, the result will be unreliable. To calculate the size of the audience, use online calculator, for instance, Optimizely:

• Baseline Conversion Rate is the average expected conversion rate calculated on the basis of previous newsletters.

• Minimum Detectable Effect is the difference between A and B.

• Statistical Significance is the confidence level characterizes the probability that the sample will be relevant for the selected results.

Optimizely calculator

You can select the audience for the newsletter in the Subscribers section. Select a resource to which your customers should be subscribed, and then select a profile database or a segment.

Next, you need to decide which percentage of this audience will participate in the test. Let's take 15%, which is approximately 305 subscribers. It means that about 152 people will receive one variation of the email, and the other 152 subscribers will receive the second version. When it is clear which template is most effective, it will be automatically sent to the remaining 1733 subscribers.

Click «Next».

Step 5. Select templates

Add previously prepared templates to the newsletter by clicking on the + button.

Step 6. Select the way to determine the winning template

There are several ways available to you to determine the most effective template:

• Number of opened messages: the platform will automatically select the template from emails that were opened more often.

• Number of message link clicks: the platform will automatically select the template in which test subscribers clicked on links more often.

• Select manually: you will manually select the template based on analytical data.

Selection winner by openings/clicks

If you want to select the winning template automatically, set the campaign Start time. This is the time and date for the mailing test launch.

After launch, the campaign will automatically collect statistics by the total number of messages openings or link clicks. You can set a time for collecting statistics using Time to select the winner parameter.

After the campaign collects statistics on opens or clicks, the message template that is most effective based on the selected indicator will be sent to the main part of the audience according to the specified schedule:

Manual winner selection

If you want to use the Select manually option, set the Start time. Campaign start is the time and date of launch of the test part of the mailing.

Next, set the campaign schedule. According to this schedule, the winner template will be sent to the main part of the audience.

Save the campaign to activate it. After the campaign sends test messages, it will switch to Active (Waiting for you) status.

At this step, you need to analyze the performance of each template and choose the winner template. To do this, click on the «Make a decision» button in the campaign editing window:

You can also select the winning template in the window with the list of campaigns by clicking on the button in the context menu:

You will see a window with statistics for each template:

Send is the number of messages sent.

Delivery rate is the number of messages delivered.

Content CTR is the percentage of clicks in a message. Calculated using the formula: (Number of clicks / Number of messages delivered) * 100%.

Content OR is the percentage of openings in a a message. Calculated using the formula: (Number of messages opened / Number of messages delivered) * 100%.

Select the template that you consider the most effective, and click the «Select» button. This template will be sent to the main part of the audience according to the specified schedule.

Step 7. Run the test

If you have not activated the campaign using the toggle switch in the main settings, do it now using the start icon on the right toolbar:

Step 8. Make a decision

You’ll reach this stage only if you’ve chosen Select manually for the winner template. Click on the corresponding icon on the right toolbar:

You will see a modal window where you can select the desired template. The data from the summary report will help you make a decision.

Conclusion

A/B/n testing allows marketers to compare several variations of one email template to send the most effective one to the bulk of subscribers. Remember that to get reliable results, it is better to test only one element of the template. Besides, you should have a sufficiently large sample of the audience.