Clicking the
Versions may vary by subject, sender, or content..
AB test settings:
- Percentage of recipients to include in the test group – Set the size of your test group. There is no "ideal" test sample size, you always must consider both the current recipient list size and the current campaign content. Always consider how large a list of recipients you have and how many recipients will receive the test sample. For example, if you have a recipient database with 1000 emails and select 40 % of them as a test sample, the campaign will be sent to 400 recipients in the first phase (version A will be sent to 200 emails, version B will also be sent to 200 emails ) and after evaluation of the test samples, the winning version will be sent to the rest of the recipient list, in this case to 600 email addresses.
- Test group evaluation period before final delivery – Set the time needed to evaluate the success of both variants. The minimum time is 10 minutes, maximum time is 4 days. When setting the time, always consider the total number of recipients and the criterion according to which success will be evaluated (e.g. Opens can be evaluated sooner than conversions)
- Evaluate the test group results by:
- Opens – Which version will be sent in the final delivery will be decided by the number of recipients opening each version of the campaign
- Visits – The number of visits for each version will decide which version will be used for the final delivery
- Clickthrough rate – The ratio between clicks and open messages and for each version will decide which version will be sent in the final distribution
- Conversion rate – The conversion rate for each version will decide which version will be used for the final delivery. Please keep in mind that the conversion code MUST be implemented on your website to measure the conversions correctly. It is necessary to set sufficient time (at least 1 day recommended) for evaluation.
AB test evaluation:
If a campaign is sent with an AB test, you will also find a detailed AB test evaluation in the campaign delivery report.
In the “DELIVERY SETTINGS” overview table you will find basic information about the AB test settings and its progress.:
Click on the red “A|B” button in the upper right corner of the “DELIVERY SETTINGS” table to view the detailed AB test evaluation:
The first part of the detailed report (TEST GROUP STATISTICS) summarizes the AB test settings again (how big was the test group, when and based on what parameter the first phase of the test was evaluated).
The second part (EVALUATION RESULTS) shows the results at the time of evaluation of the first phase of the AB test, i.e. the results just before the moment when the winning variant was sent.
The last part (RUNNING RESULTS) shows the current development of metrics for recipients from the test group - in other words, what the evaluation of the initial phase of the AB test would look like at the moment you view the report, i.e. if the first phase of the AB test had run for a longer period.
As can be seen from the image above - if the first phase of the AB test had lasted longer (the evaluation would have taken place later), variant B would have been sent as the winner.
The “flame” icon (see orange arrows in the image above) indicates that for a given metric, there is a significantly greater difference in the results of individual variants compared to the metric set for AB test evaluation, and therefore this metric should be used for evaluation next time.
