--- title: "Create a holdout group" slug: "create-a-holdout-group" updated: 2026-11-17T14:53:35Z published: 2026-11-17T14:53:35Z canonical: "docs.xtremepush.com/create-a-holdout-group" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.xtremepush.com/llms.txt > Use this file to discover all available pages before exploring further. # Create a holdout group A holdout group is a random group of users who do not receive a loyalty feature, such as a quest. Comparing the holdout group with everyone else in your loyalty program shows you what the feature really adds. For example, many users might complete a quest, but they might have deposited or played just as much without it. If users who can receive the quest deposit more, log in more often, or stay active for longer than the holdout group, the quest is making a difference. If both groups behave the same, it is not. To create a holdout group, you give every user a random group number from 1 to 10 in a profile attribute. You then restrict the quest so that users in one group never receive it. Because the group number is stored in a profile attribute, it syncs to the user's Xtremepush profile automatically. You can then compare the holdout group with other users in Xtremepush segments. ## Steps to holdout user from a quest This example keeps a random 10% of users out of a quest, using a `holdout_group_1` attribute. To create a holdout group, do the following: 1. In Xtremepush, go to **Data** > **Data Manager** and create a number tag named `holdout_group_1` with **Profile Attribute** enabled. The `holdout_group_1` attribute then appears in the Loyalty Hub under **Users** > **Attributes**. > [!TIP] > Use a separate attribute for each holdout test. If two tests share an attribute, the same users are held out of both tests, and you cannot tell which feature caused a change in their behaviour. 2. In the Loyalty Hub, add an automation that gives every new user a random group number: 1. Go to **Users** > **Attributes**, select `holdout_group_1`, click **Manage**, and then click **Add Automation**. 2. Select **Event-based**, then select `USER_CREATED` under **Internal Events** as the event. This triggers the automation once, when the user's loyalty account is created. > [!TIP] > The event list can show `USER_CREATED` twice: under **Custom Events Sent by You**, which are events your platform sends through the API, and under **Internal Events**, which are events the Loyalty Hub generates. Select the one under **Internal Events**. 3. Enter `randomInt(1, 11)` as the expression and click **Submit**. This gives the user a random whole number from 1 to 10, with each number equally likely. `randomInt` does not include the upper value, so you enter 11 to get numbers up to 10. Each group holds roughly 10% of your users. In step 4, you choose which groups are kept out of the quest, which sets the size of the holdout group. For example, keeping out group 1 holds out about 10% of users. ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/holdout-step1c-event-expression-randomint.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A57%3A40Z&se=2026-11-22T19%3A57%3A40Z&sr=b&sp=r&sig=XIYIJsgx0itRRLfJgs4Mg9CaAzWEW%2FXsPuipQtvyOpk%3D) 3. Add a one-off automation to give existing users a group number. The automation from step 2 only runs when a loyalty account is created, so existing users do not have a group number, and a quest restricted by group number is hidden from them. To assign their group numbers, create a one-off schedule-based automation that runs once for all matching users. ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/holdout-step2b-backfill-cron-filter-expression.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A57%3A40Z&se=2026-11-22T19%3A57%3A40Z&sr=b&sp=r&sig=zL%2FE1uptDuaqnk9pUrXO7P4u9doqBCz4YiBuArPyxRs%3D) 1. Go to **Users** > **Attributes**, select `holdout_group_1`, click **Manage**, and then click **Add Automation**. 2. Select **Schedule-based** and enter any cron expression. A new schedule-based automation runs at the next full hour, whatever the cron expression says. 3. Under **User Filters**, add `holdout_group_1` **empty**. This limits the automation to users who do not have a group number yet, so no user's group changes. 4. Enter `randomInt(1, 11)` as the expression and click **Submit**. 5. After the next full hour, wait for the automation to finish, and check under **Users** > **All Users** that no users have an empty `holdout_group_1`. 6. Delete the one-off automation you added in step 3. You only need it once, to catch up existing users. 4. Keep the holdout group out of the quest: 1. Go to **Quests**, edit the quest, and go to **Restrictions** > **Additional restrictions**. 2. Click **Add Requirement** and select `holdout_group_1`. 3. Select **>**, enter `1`, and click **Submit**. Only users in groups 2 to 10 can now receive the quest. There is no "not equal" operator, so `> 1` is how you exclude group 1. Repeat this step for every quest that is part of the test. ![](https://cdn.document360.io/b056e88b-f6c1-4bd2-b5b8-08e9336a17c9/Images/Documentation/holdout-quest-4-requirement-gt-1.png?sv=2026-02-06&spr=https&st=2026-11-22T18%3A57%3A40Z&se=2026-11-22T19%3A57%3A40Z&sr=b&sp=r&sig=HkYApsd3cJTy2ueK7naB%2FRcCmIacfkIaSDnl1vcqJ4Y%3D) To measure the results, create a segment in Xtremepush for users with `holdout_group_1` = `1`, and compare their deposits, activity, and retention with the rest of your users. For a larger holdout group, raise the value. For a 20% holdout group, use `holdout_group_1` > `2` on the quest and `holdout_group_1` ≤ `2` for the segment. For 30%, use > `3` and ≤ `3`. > [!WARNING] > Do not edit the automations or their expression while a test is running. If new users get group numbers from a different range, the groups are no longer equal in size, and the results are harder to compare.