Multivariate Testing: What It is & How to Conduct It

Multivariate testing (or A/B/n testing) allows you to compare the effectiveness of several content variations in email newsletters and other mailings. It is possible to test templates as a whole or their individual elements. The goal is to understand which version works better and brings more conversions.

Specifically, A/B/n can help you test the performance of:

the amount of text in emails;
the text, design, size or placement of CTA buttons;
the subject line or preheader of an email newsletter;
the title or icon of a push notification;
product prices;
product descriptions and headings within the description;
product illustrations.

What Makes A/B/n Testing Helpful

Reducing the risk of financial loss. The easier it is for a customer to find the right product and place an order, the more likely they are to make a purchase. Inconspicuous buttons and confusing navigation lead to loss of customers and potential revenue.
Increasing conversion. The test helps identify the most clickable options that will lead your customers to purchase faster.
Testing ideas. If disputes arise during the development stage, split testing allows you to check whose idea or hypothesis turned out to be more effective.
Saving time. The test allows you to identify effective options faster than sequentially testing several content versions one by one.
New ideas for future campaigns. User reactions to different content options can serve as a source of new insights (sometimes unexpected).

How to Conduct A/B/n Testing

In Altcraft Platform, both simple split tests (A/B) and more complex A/B/n testing are available. As an example, we will conduct a simple A/B test: compare two variations of a single message.

Step 1. Identify the Problem and Formulate a Hypothesis

The first step is to identify the weak points of your marketing campaigns. This can be done with the help of analytical tools. For example, you can use the Altcraft platform's channel report, which collects key metrics of messages across different channels.

Suppose you notice that a welcome email newsletter has a high Open Rate but a low CTR. This means the subject line and preheader are good, and the problem most likely lies with the content: irrelevant text, weak visuals, or an ineffective call to action.

Another useful tool is the click map. This is a visual representation of your email showing the number of unique clicks next to each link. The click map can be used to determine the most successful placement and design of links.

Step 2. Create Template Variations for the Test

Next, you need to create several template options. You can test up to 8 templates simultaneously.

Important: the more message variants you have, the larger the test audience must be. Results will be meaningful only with sufficient traffic for each template option. Also, to obtain reliable results, it is recommended to test only one template element at a time.

Go to the «Templates» section. Click the «Create» button at the top of the page. Select the method (via import, using code, or in the editor) and the desired sending channel (Email, SMS, Push, Notify, Max, Telegram, Viber, WhatsApp). Then you can start creating the template.

We will test two email templates for a promotional newsletter. The goal is to check which one will prove more effective.

Step 3. Create the Mailing

In the «Mailings» section, select «Multivariate testing». Click the «Create» button at the top of the page to open the settings.

Enter the mailing name, a short description, tags for quick search, and access groups. If you wish to launch the mailing as soon as the date specified in the schedule arrives, click the «Active» toggle switch in the upper right corner.

Basic settings of an MVT newsletter

In the section below, select the channel and click «Next».

Sending channel

Step 4. Determine the Sample Size

Next, you need to determine how many subscribers will participate in the experiment. To calculate the appropriate audience size, use an online calculator, for example, Optimizely:

Baseline Conversion Rate — the average expected conversion rate (taking into account experience from previous mailings).
Minimum Detectable Effect — the difference between A and B that will be noticeable to you.
Statistical Significance — the confidence level. It characterises the probability that the sample will be significant for the selected results.

Optimizely calculator

The audience for the newsletter is defined in the «Subscribers» section. Select the resource that customers are subscribed to, and then the profile database or segment.

Next, you need to select what percentage of this audience will participate in the experiment. In our example, we will select 20%: one half of them will receive the first template variant, and the other half the second. After the most effective option is determined, the winning template will be automatically sent to the remaining subscribers.

Click «Next».

Step 5. Add Templates

Add the required message templates by clicking the + button.

Step 6. Select the Method for Determining the Winner Template

The most effective template can be determined by:

the number of opened messages (the platform will automatically select the template that was opened more often);
the number of message link clicks (the platform will automatically select the template in which test subscribers clicked on links more often);
user decision (you select the template manually, based on analytical data).

Selecting the winner template by opens/clicks

Specify the time and date of the mailing launch in the «Start time» field.

After launching the mailing in the automatic winner selection mode, statistics on the total number of opens or link clicks in the templates will be collected automatically. The time for collecting analytics is set using the «Time to select the winner» parameter.

After the open or click statistics have been collected, the most effective message template (according to the selected metric) will be sent to the rest of the audience by the specified schedule.

Selecting the winner template manually

If the winner is selected manually, you will need to set the start time and the mailing schedule. The winner template will be sent to the remaining part of the audience according to this schedule.

Click «Finish», check the newsletter parameters and save it. After the test messages are sent, the status will change to «Active (Waiting)».

Next, you need to analyze the metrics of each template and select the winner template. To do this, click the «Make decision» button in the mailing editing window:

The winning template can also be selected in the newsletter list by clicking the corresponding button in the context menu.

In the window that shows with statistics for each template, the following will be displayed:

Send — the number of messages sent.
Delivery rate — the number of messages delivered.
Content CTR — the percentage of clicks in the message, calculated using the formula: (Number of clicks / Number of delivered messages) × 100%.
Content OR — the percentage of message opens, calculated using the formula: (Number of opens / Number of delivered messages) × 100%.

Select the most effective message and click the «Select» button.

Step 7. Launch the Test

If you did not click the «Active» toggle switch during the mailing setup, you can do it now by clicking the launch icon on the right panel.

Step 8. Make a Decision

You will only reach this step if you set to choose the winner template manually.

Click the «Select winner message» icon on the right panel. A window will appear where you can manually select the desired template. The data from the channel report will help you make a decision.