A/B Testing
Last updated
Last updated
A/B Testing is a way to compare notifications to determine which one performs better. A/B testing is usually used to determine the effectiveness of message content, for example, to understand whether a HoloPush notification performs better than a Rich Media notification.
To create an A/B test, navigate to the A/B testing section of the 2mee platform then click on the 'Create A/B test Message' button.
When you create an A/B test you will need to give this a name. The name of your A/B test will be used when listing the message on the 2mee platform, for example, to identify the message when viewing reports.
You will now be asked to add a target audience for your A/B test message, please choose an audience segment or upload a CSV file.
The A/B testing calculation we use is; 25% of the segment (the segment must have a minimum 8 users) is chosen as a test group (The 25% can only be a maximum of 8000 users), this is then split into A and B test groups which are 12.5% of the total audience each. The two test messages are sent to these groups of 12.5% then you can select which is the winning message. The winning message will then be sent to the remaining 75% of the audience.
Now you have chosen your audience you need to add all of the details for both message variant A and B. Message templates are the best way to quickly send A/B messages, if you do not have a template created already you can create either a HoloPush notification or Text push notification.
Once you have chosen your message type you will need to add all of the message details for both message variant A and B.
The quickest way to create A/B test messages is to use pre made message templates
Once you have added all of the details for both your A/B test message you will need to set an expiry time for these messages. After setting your expiry time you can choose to send your A/B test messages immediately or choose a specific time/date to start sending the message.
After you have scheduled an A/B Test, you can view the reporting for both A/B variants by going to the A/B testing section and using the dropdown to select sent messages. Once you have chosen to view sent messages, click the message report you would like to view.
In the A/B test report, you can see key metrics like Clicks, Clickthrough Rate, and Deliveries for both variants. These metrics can be used to determine which message performed best and send that winning message variant to the rest of your target audience.
After you have viewed the reports for both message variants on the 2mee platform you can choose which is the winning message to send to the rest of your audience. 2mee provides key metrics to help you make your decision on which message is the winner, these are Clicks and the Clickthrough Rate.
Once you have chosen the winning variant we will send this immediately to the remaining users of your target audience.
Any users who received variant A or variant B will not receive the winning message variant.
You will be able to find the report for the winning messages in the HoloPush sent messages section or by clicking the 'View winning message report' button on the sent A/B test.
You can also get detailed reports for each individual variant by clicking the view report button shown below.
For a complete overview of how to create an A/B test, take a look at this video from our Product Manager; Dan Kersley.
