An A/B test shows different groups of users different versions of a loyalty feature, such as a quest, so you can find out which version performs better. For example, you have a quest that rewards users with 10 tokens. You want to know whether doubling the reward to 20 tokens makes more users complete the quest, and whether those users then deposit or play more, enough to justify the extra tokens.
An A/B test answers this by showing half your users the 10-token version and the other half the 20-token version, then comparing the two groups.
To run an A/B test, you give every user a random group number in a profile attribute. The example on this page uses group numbers from 1 to 10.
You then create one version of the quest for each variant and restrict each version to a range of group numbers, so each user only receives one version. Because the group number syncs to the user's Xtremepush profile automatically, you can also compare the groups in Xtremepush segments.
How group numbers work
Group numbers do not limit how many users take part. The example on this page uses group numbers from 1 to 10. Each number is a group, and every user is placed in one group at random. For example, with 1,000 users, roughly 100 users land in each group, so Variant A (groups 1 to 5) reaches about 500 users and Variant B (groups 6 to 10) reaches the other 500.
Because the groups are random, each one has a similar mix of users, so any difference between the variants comes from the feature you are testing.
Steps to A/B test a loyalty feature
This example tests whether a quest performs better with a reward of 10 tokens or 20 tokens. It splits users 50/50 using an ab_test_group_1 attribute.
To A/B test a loyalty feature, do the following:
In Xtremepush, go to Data > Data Manager and create a number tag named
ab_test_group_1with Profile Attribute enabled.The
ab_test_group_1attribute then appears in the Loyalty Hub under Users > Attributes.Use a separate attribute for each A/B test. If two tests share an attribute, the same users end up in the same group for both tests, and you cannot tell which test caused a change in their behaviour.
In the Loyalty Hub, add an automation that gives every new user a random group number:
Go to Users > Attributes, select
ab_test_group_1, click Manage, and then click Add Automation.Select Event-based, then select
USER_CREATEDunder Internal Events as the event.This triggers the automation once, when the user's loyalty account is created.
The event list can show
USER_CREATEDtwice: under Custom Events Sent by You, which are events your platform sends through the API, and under Internal Events, which are events the Loyalty Hub generates. Select the one under Internal Events.Enter
randomInt(1, 11)as the expression and click Submit.This gives the user a random whole number from 1 to 10, with each number equally likely.
randomIntdoes not include the upper value, so you enter 11 to get numbers up to 10. Each group holds roughly 10% of your users.In step 4, you choose which groups receive each version of the quest, which sets the split. For example, giving one version groups 1 to 5 and the other groups 6 to 10 splits users 50/50.

Add a one-off automation to give existing users a group number.
The automation from step 2 only runs when a loyalty account is created, so existing users do not have a group number. To assign their group numbers, create a one-off schedule-based automation that runs once for all matching users.
Go to Users > Attributes, select
ab_test_group_1, click Manage, and then click Add Automation.Select Schedule-based and enter any cron expression. A new schedule-based automation runs at the next full hour, whatever the cron expression says.
Under User Filters, add
ab_test_group_1empty. This limits the automation to users who do not have a group number yet, so no user's group changes.Enter
randomInt(1, 11)as the expression and click Submit.After the next full hour, wait for the automation to finish, and check under Users > All Users that no users have an empty
ab_test_group_1.Delete the one-off automation you added in step 3. You only need it once, to catch up existing users.

Create two versions of the quest you want to test. For more information, see Configure Quests.
In each version, go to Restrictions > Additional restrictions and add a requirement on
ab_test_group_1, so only one range of groups receives it.In this example, the two versions differ as follows:
Variant A: Requirement
ab_test_group_1≤5, so only users in groups 1 to 5 receive it. The reward is 10 tokens.Variant B: Requirement
ab_test_group_1>5, so only users in groups 6 to 10 receive it. The reward is 20 tokens.
Variant A
Variant B


To compare quest performance, click the stats icon on each version to see how many users received, accepted, and completed it. To compare business impact, create segments in Xtremepush for users with ab_test_group_1 ≤ 5 and ab_test_group_1 > 5, and compare their deposits, activity, and retention.
To split users differently, change the values. For a 30/70 split, use ≤
3and >3. For three versions split 30/30/40, use ≤3, then >3and ≤6, then >6. To run the test with a holdout group, click Add Another Requirement on each version and addholdout_group_1>1.