> ## Documentation Index
> Fetch the complete documentation index at: https://docs.notifly.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# A/B testing

> Send different messages to groups of users and compare how each message performs.

## 1. Set up an A/B test campaign

### 1-1. Choose the campaign type

In **Basic Information and Channel**, select **A/B Test Campaign** to open the A/B test settings.

### 1-2. Configure the messages and actions

Under **Message and Action**, set a title, message, and image for each group. Notifly randomly assigns users to groups based on the proportions you enter.

![Message variants and allocation in an A/B test](https://mintcdn.com/notifly/dcFO7QF5JScPffiH/images/i18n/en/campaigns/abtest-message.png?fit=max\&auto=format\&n=dcFO7QF5JScPffiH\&q=85\&s=ceea1315507fb733757f658a5bfdd6d8)

> [!NOTE]
>
> Add a **Control Group** to compare results for users who receive a message with results for users who receive none.

In **Analysis Configuration**, add at least one event under **Conversion Events** before creating the A/B test campaign.

## 2. End an A/B test

In the campaign list, select **End A/B Test**, then choose which group to keep. If you choose a group that received a message, the campaign becomes a **Standard Campaign** and uses that message for the entire target audience. If you choose the **Control Group**, the campaign ends.

![Choose the winning A/B test group](https://mintcdn.com/notifly/dcFO7QF5JScPffiH/images/i18n/en/campaigns/abtest-winner.png?fit=max\&auto=format\&n=dcFO7QF5JScPffiH\&q=85\&s=e8331f81099dbb606e07bf06f71974c2)

## 3. Analyze an A/B test

Open **Campaign Delivery Status** to see daily test results. Under **A/B test results**, select the event in **Event name** and the two groups in **Comparison group**.

![Daily A/B test results by group](https://mintcdn.com/notifly/dcFO7QF5JScPffiH/images/i18n/en/campaigns/abtest-report.png?fit=max\&auto=format\&n=dcFO7QF5JScPffiH\&q=85\&s=d34b4011e350f55f1f932fd95059060a)

* **Total events / Total users** shows the selected event count and the number of potential recipients in each group. The number of actual recipients can be lower. The value in parentheses is the event count divided by the user count.
* **Improvement vs. baseline** is the percentage change in events per user compared with the baseline group.
* **p-value** helps you assess whether the difference between a comparison group and the baseline is statistically significant. Values range from 0 to 1. A p-value below 0.05 is commonly considered statistically significant.
* **Confidence interval** gives the range of plausible values for lift over the baseline. A 95% confidence interval that excludes zero indicates a statistically significant difference at the 5% level.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.