A/B testing in a mailing

Why do I need A/B tests in mailing?

You can perform A/B testing in a mass mailing. That is, create different versions of the message and check which one gives the best results. For example, you can evaluate how different messages affect the amount of receipt or the number of receipts on average per customer.

A/B-testing can be done via e-mail, SMS or push mailing.

By default, one variant of the message is sent to all customers in a mass mailing. The A/B-test option allows you to send from 2 to 5 different message variants within one mailing.

The messages can be changed in different variants:

• Message body;

• Sender;

• The subject of the message (for e-mail messages);

• Attached files (for e-mail messages).

An example of A/B testing in a mailing

You can create 2 variants of the message in the SMS mailing:

• "50% off on all natural uggs."

• "Up to 50% discount for you on uggs made of natural materials".

Creating an A/B test in a mass mailing

To add a new variant of a message to the mailing, go to the A/B test block on the mailing list creation page. This block is located between the Customers and Mailing blocks.

Click Add variant.

Adding a new variant of the A/B test

For each message variant, you can set what % of customers should receive it. By default, the variants are distributed evenly. For example, if there are two variants, half of the customers will receive the first variant, and half will receive the second one.

You can change the percentage ratio by moving the slider:

Here is an example of an uneven distribution of responses:

• Variant A will get 15% of customers;

• Variant B - 25%;

• Variant C - 35%;

• Variant D - 25% of customers.

When creating an A/B test in a mass mailing, tabs appear in the Mailings block. The number of tabs corresponds to the number of testing variants. Clicking on the name of a tab takes you to filling out the message parameters. It is necessary to fill out these parameters for each variant.

You can delete a test option by clicking on the trash can icon in the corresponding tab.

After filling out all the message variants, you can send a test message with the desired variant. To do this, select the tab with the desired message variant and click Test:

To preview an email, select the tab with the required variant of the message and click the preview button:

By default, for sending a test message and during previewing Option A will be used.

Reports

The A/B test report is available on the page where all the other mailings reports are located.

Report looks like this:

Control group and A/B tests report

The first table displays information for each group:

• Number of recipients: how many messages were sent to that group;

• Messages delivered: how many messages were delivered to that group;

• Number of customers, who made a purchase, is calculated considering attribution parameters;
• Conversion into customers: how many percent of customers, who received the message, made a purchase under the influence of the mailing;

• Number of receipts in the group;

• Average number of receipts per customer;

• Amount of purchases;

• Average revenue per customer.

The second table displays detailed information about conversion and undesirable behavior in each group. For example, in group B:

• Sent: 36,057 messages (100%);

• Delivered: 36,057 messages (100% of messages sent);

• Opened: 918 messages (2.5% of messages sent);

• Transfers came from 198 messages (0.5% of sent messages);

• Orders: from 26 messages (0.1% of sent messages);

• 2 persons unsubscribed after receiving a message (0%);

• Spam complaints: 0 (0%).

The values in some columns are highlighted in color: green - the best value, red - the worst.

For example, Group A has an average revenue per customer of 25 USD, which is the best value among all the groups (highlighted in green). In group B - 10 USD, this is the worst value (highlighted in red).

The report shows the attributed receipts associated with the mailing. You can read more about attribution in the Check attribution article.