Skip to main content

1. Set up an A/B test campaign

1-1. Choose the campaign type

In Basic Information and Channel, select A/B Test Campaign to open the A/B test settings.

1-2. Configure the messages and actions

Under Message and Action, set a title, message, and image for each group. Notifly randomly assigns users to groups based on the proportions you enter. Message variants and allocation in an A/B test
Add a Control Group to compare results for users who receive a message with results for users who receive none.
In Analysis Configuration, add at least one event under Conversion Events before creating the A/B test campaign.

2. End an A/B test

In the campaign list, select End A/B Test, then choose which group to keep. If you choose a group that received a message, the campaign becomes a Standard Campaign and uses that message for the entire target audience. If you choose the Control Group, the campaign ends. Choose the winning A/B test group

3. Analyze an A/B test

Open Campaign Delivery Status to see daily test results. Under A/B test results, select the event in Event name and the two groups in Comparison group. Daily A/B test results by group
  • Total events / Total users shows the selected event count and the number of potential recipients in each group. The number of actual recipients can be lower. The value in parentheses is the event count divided by the user count.
  • Improvement vs. baseline is the percentage change in events per user compared with the baseline group.
  • p-value helps you assess whether the difference between a comparison group and the baseline is statistically significant. Values range from 0 to 1. A p-value below 0.05 is commonly considered statistically significant.
  • Confidence interval gives the range of plausible values for lift over the baseline. A 95% confidence interval that excludes zero indicates a statistically significant difference at the 5% level.