Create a holdout group

Prev Next

A holdout group is a random group of users who do not receive a loyalty feature, such as a quest. Comparing the holdout group with everyone else in your loyalty program shows you what the feature really adds.  

For example, many users might complete a quest, but they might have deposited or played just as much without it. If users who can receive the quest deposit more, log in more often, or stay active for longer than the holdout group, the quest is making a difference. If both groups behave the same, it is not.

To create a holdout group, you give every user a random group number from 1 to 10 in a profile attribute. You then restrict the quest so that users in one group never receive it. Because the group number is stored in a profile attribute, it syncs to the user's Xtremepush profile automatically. You can then compare the holdout group with other users in Xtremepush segments.

Steps to holdout user from a quest

This example keeps a random 10% of users out of a quest, using a holdout_group_1 attribute.

To create a holdout group, do the following:

  1. In Xtremepush, go to Data > Data Manager and create a number tag named holdout_group_1 with Profile Attribute enabled.

    The holdout_group_1 attribute then appears in the Loyalty Hub under Users > Attributes.

    Use a separate attribute for each holdout test. If two tests share an attribute, the same users are held out of both tests, and you cannot tell which feature caused a change in their behaviour.

  2. In the Loyalty Hub, add an automation that gives every new user a random group number:

    1. Go to Users > Attributes, select holdout_group_1, click Manage, and then click Add Automation.

    2. Select Event-based, then select USER_CREATED under Internal Events as the event.

      This triggers the automation once, when the user's loyalty account is created.

      The event list can show USER_CREATED twice: under Custom Events Sent by You, which are events your platform sends through the API, and under Internal Events, which are events the Loyalty Hub generates. Select the one under Internal Events.

    3. Enter randomInt(1, 11) as the expression and click Submit.

      This gives the user a random whole number from 1 to 10, with each number equally likely. randomInt does not include the upper value, so you enter 11 to get numbers up to 10. Each group holds roughly 10% of your users. In step 4, you choose which groups are kept out of the quest, which sets the size of the holdout group. For example, keeping out group 1 holds out about 10% of users.

  3. Add a one-off automation to give existing users a group number.

    The automation from step 2 only runs when a loyalty account is created, so existing users do not have a group number, and a quest restricted by group number is hidden from them. To assign their group numbers, create a one-off schedule-based automation that runs once for all matching users.

    1. Go to Users > Attributes, select holdout_group_1, click Manage, and then click Add Automation.

    2. Select Schedule-based and enter any cron expression. A new schedule-based automation runs at the next full hour, whatever the cron expression says.

    3. Under User Filters, add holdout_group_1 empty. This limits the automation to users who do not have a group number yet, so no user's group changes.

    4. Enter randomInt(1, 11) as the expression and click Submit.

    5. After the next full hour, wait for the automation to finish, and check under Users > All Users that no users have an empty holdout_group_1.

    6. Delete the one-off automation you added in step 3. You only need it once, to catch up existing users.

  4. Keep the holdout group out of the quest:

    1. Go to Quests, edit the quest, and go to Restrictions > Additional restrictions.

    2. Click Add Requirement and select holdout_group_1.

    3. Select >, enter 1, and click Submit.

      Only users in groups 2 to 10 can now receive the quest. There is no "not equal" operator, so > 1 is how you exclude group 1.

      Repeat this step for every quest that is part of the test.

To measure the results, create a segment in Xtremepush for users with holdout_group_1 = 1, and compare their deposits, activity, and retention with the rest of your users.

For a larger holdout group, raise the value. For a 20% holdout group, use holdout_group_1 > 2 on the quest and holdout_group_1 ≤ 2 for the segment. For 30%, use > 3 and ≤ 3.

Do not edit the automations or their expression while a test is running. If new users get group numbers from a different range, the groups are no longer equal in size, and the results are harder to compare.