--- title: "A/B test a loyalty feature" slug: "ab-test-a-loyalty-feature" updated: 2026-11-17T14:53:29Z published: 2026-11-17T14:53:29Z canonical: "docs.xtremepush.com/ab-test-a-loyalty-feature" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.xtremepush.com/llms.txt > Use this file to discover all available pages before exploring further. # A/B test a loyalty feature An A/B test shows different groups of users different versions of a loyalty feature, such as a quest, so you can find out which version performs better. For example, you have a quest that rewards users with 10 tokens. You want to know whether doubling the reward to 20 tokens makes more users complete the quest, and whether those users then deposit or play more, enough to justify the extra tokens. An A/B test answers this by showing half your users the 10-token version and the other half the 20-token version, then comparing the two groups. To run an A/B test, you give every user a random group number in a profile attribute. The example on this page uses group numbers from 1 to 10. You then create one version of the quest for each variant and restrict each version to a range of group numbers, so each user only receives one version. Because the group number syncs to the user's Xtremepush profile automatically, you can also compare the groups in Xtremepush segments. > [!NOTE] > How group numbers work > > Group numbers do not limit how many users take part. The example on this page uses group numbers from 1 to 10. Each number is a group, and every user is placed in one group at random. For example, with 1,000 users, roughly 100 users land in each group, so Variant A (groups 1 to 5) reaches about 500 users and Variant B (groups 6 to 10) reaches the other 500. > > > > Because the groups are random, each one has a similar mix of users, so any difference between the variants comes from the feature you are testing. ## Steps to A/B test a loyalty feature This example tests whether a quest performs better with a reward of 10 tokens or 20 tokens. It splits users 50/50 using an `ab_test_group_1` attribute. To A/B test a loyalty feature, do the following: 1. In Xtremepush, go to **Data** > **Data Manager** and create a number tag named `ab_test_group_1` with **Profile Attribute** enabled. The `ab_test_group_1` attribute then appears in the Loyalty Hub under **Users** > **Attributes**. > [!TIP] > Use a separate attribute for each A/B test. If two tests share an attribute, the same users end up in the same group for both tests, and you cannot tell which test caused a change in their behaviour. 2. In the Loyalty Hub, add an automation that gives every new user a random group number: 1. Go to **Users** > **Attributes**, select `ab_test_group_1`, click **Manage**, and then click **Add Automation**. 2. Select **Event-based**, then select `USER_CREATED` under **Internal Events** as the event. This triggers the automation once, when the user's loyalty account is created. > [!TIP] > The event list can show `USER_CREATED` twice: under **Custom Events Sent by You**, which are events your platform sends through the API, and under **Internal Events**, which are events the Loyalty Hub generates. Select the one under **Internal Events**. 3. Enter `randomInt(1, 11)` as the expression and click **Submit**. This gives the user a random whole number from 1 to 10, with each number equally likely. `randomInt` does not include the upper value, so you enter 11 to get numbers up to 10. Each group holds roughly 10% of your users. In step 4, you choose which groups receive each version of the quest, which sets the split. For example, giving one version groups 1 to 5 and the other groups 6 to 10 splits users 50/50. ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/ab-step2a-2-event-expression-randomint.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A39%3A50Z&se=2026-11-22T19%3A39%3A50Z&sr=b&sp=r&sig=tnK7qSZXpcDI3w7nNxv55zC58itmwpzV3LcT%2BK6k2AM%3D) 3. Add a one-off automation to give existing users a group number. The automation from step 2 only runs when a loyalty account is created, so existing users do not have a group number. To assign their group numbers, create a one-off schedule-based automation that runs once for all matching users. ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/ab-step2b-2-backfill-cron-filter-expression.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A39%3A50Z&se=2026-11-22T19%3A39%3A50Z&sr=b&sp=r&sig=6nX8C6SnzqOs%2F0ZrH5%2BadlRs%2FxGWjhNziZrAXsROKW0%3D) 1. Go to **Users** > **Attributes**, select `ab_test_group_1`, click **Manage**, and then click **Add Automation**. 2. Select **Schedule-based** and enter any cron expression. A new schedule-based automation runs at the next full hour, whatever the cron expression says. 3. Under **User Filters**, add `ab_test_group_1` **empty**. This limits the automation to users who do not have a group number yet, so no user's group changes. 4. Enter `randomInt(1, 11)` as the expression and click **Submit**. 5. After the next full hour, wait for the automation to finish, and check under **Users** > **All Users** that no users have an empty `ab_test_group_1`. 6. Delete the one-off automation you added in step 3. You only need it once, to catch up existing users. 4. Create two versions of the quest you want to test. For more information, see [Configure Quests](/documentation/docs/configure-quests). In each version, go to **Restrictions** > **Additional restrictions** and add a requirement on `ab_test_group_1`, so only one range of groups receives it. In this example, the two versions differ as follows: | Variant A | Variant B | | --- | --- | | ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/ab-variant-a-requirement-le5-reward-10.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A39%3A50Z&se=2026-11-22T19%3A39%3A50Z&sr=b&sp=r&sig=c9YYuPm64CsFU0xI8YHf3SCImYQiYDSVZWHht2ltuSQ%3D) | ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/ab-variant-b-requirement-gt5-reward-20.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A39%3A50Z&se=2026-11-22T19%3A39%3A50Z&sr=b&sp=r&sig=XQwO%2B%2FdRGMlF6dAxf%2FhcBO%2FY22l5ujU1nAHvqqPfiZw%3D) | - **Variant A**: Requirement `ab_test_group_1` ≤ `5`, so only users in groups 1 to 5 receive it. The reward is 10 tokens. - **Variant B**: Requirement `ab_test_group_1` > `5`, so only users in groups 6 to 10 receive it. The reward is 20 tokens. To compare quest performance, click the stats icon on each version to see how many users received, accepted, and completed it. To compare business impact, create segments in Xtremepush for users with `ab_test_group_1` ≤ `5` and `ab_test_group_1` > `5`, and compare their deposits, activity, and retention. > [!NOTE] > To split users differently, change the values. For a 30/70 split, use ≤ `3` and > `3`. For three versions split 30/30/40, use ≤ `3`, then > `3` and ≤ `6`, then > `6`. To run the test with a holdout group, click **Add Another Requirement** on each version and add `holdout_group_1` > `1`.