Kafka Integration Guide

Overview

Kafka integration allows your backend systems to stream event data and synchronise user profiles with Xtremepush at high volume and low latency. Rather than making individual API calls per event or user update, your infrastructure publishes messages to Kafka topics that Xtremepush consumes continuously.

This guide covers the two integration approaches available, the required topic formats and payload schemas, and guidance on handling high-volume data scenarios.

Before you begin

Before starting your Kafka integration, make sure you have the following in place:

• Kafka credentials — each project is provisioned with a unique username, password, and topic names. Contact Xtremepush support if you have not yet received these.
• Egress IP addresses — you will need to provide the IP addresses your infrastructure will send events from so Xtremepush can allowlist them.
• SCRAM-SHA-512 support — all connections use SCRAM-SHA-512 authentication. Confirm your infrastructure supports this before starting.
• Volume estimates — approximate event volumes for both test and production environments, so Xtremepush can scale workers appropriately.

Architecture & key concepts

Before starting your integration, you'll need to understand how Xtremepush interacts with your Kafka infrastructure.

Topics

Data is organised into topics. Each project has two corresponding topics: events and users. Unique credentials are provisioned per project.

Partitions & scaling

Topics can be separated into partitions, allowing Xtremepush to scale consumers horizontally to handle increased throughput. Messages with the same key are routed to the same partition, ensuring ordering is maintained per user.

Offsets

Each consumer maintains its own record of progress (an offset). If a connection is interrupted, Xtremepush resumes from the last acknowledged offset to ensure no data loss.

Integration approaches

There are two primary routes for establishing a Kafka connection. The chosen route determines the level of configuration required from your technical team.

Route 1: Client push to Xtremepush standard format

This is the most efficient route and requires configuration on the Xtremepush side only. Your infrastructure produces (sends) data directly to the topics hosted on our brokers in our required format.

Each project is provisioned with two topics (events and users) and a unique username and password, provided by Xtremepush support. Note that profile attributes can also be set via an event payload using the user_attributes object, in the same way as the Xtremepush API.

Authentication

Connections are established using SCRAM-SHA-512. Credentials are unique per project and provided by Xtremepush support.

What we need from you

To complete the setup, we require the following from your technical team:

• The egress IP addresses your infrastructure will be sending events from.
• Confirmation that you are able to connect via SCRAM-SHA-512 authentication.
• Expected event volumes — this determines how we scale internal workers. Please provide estimates for both test and production environments.
• Details of any bulk loading requirements — sending large volumes of messages in a short period can cause processing delays. If bulk loading is part of your use case, we need to discuss this upfront. See the note in the Users topic section for more detail.
• Where you operate multiple projects (e.g., different brands or regions), confirmation that you can produce each project's data to a separate set of topics on our side.

Route 2: Partner ecosystem (standard integrations)

If you use one of our core platform partners, the Kafka integration is largely pre-configured.

Topic formats & payload requirements

All payloads must be valid JSON. Each project has two topics, each serving a distinct purpose.

Events topic

Use this topic to record events and actions a user performs (e.g. bets, deposits, logins).

• Topic name: Defined as part of the setup
• Message key: Not required, but can be set to a unique event UUID. Xtremepush will deduplicate events within a configurable time window based on this key. To enable deduplication for your project, contact Xtremepush support.
• Message headers: None
• Message body: JSON encoded item

Schema

{
  "event": "some_event",
  "user_id": "some_user",
  "user_attributes": {
    "any_attr": "any_value"
  },
  "value": {
    "any_key": "any_value"
  },
  "timestamp": "2025-01-01 12:00:00"
}

Example

{
  "event": "bet",
  "user_id": "3e685367-07d5-4d48-93ae-f007ac336605",
  "customer_id": 12345,
  "user_attributes": {
    "customer_tier": "VIP"
  },
  "value": {
    "bet_id": "3e685367-07d5-4d48-93ae-f007ac336605",
    "odds": 12.2,
    "stake": 100.0
  },
  "timestamp": "2024-09-01 12:00:00.123"
}

Users topic

Use this topic to create new users or update existing user profiles.

🚧

Important — sequencing

Profile messages are processed sequentially and large imports may take significant time to complete. If events are sent on the events topic for users whose profiles have not yet been processed, those events will be dropped. Ensure your profile import is complete before sending events for those users, and avoid bulk loading large volumes of profile messages in a short period where possible.

• Topic name: Defined as part of the setup
• Message key: Recommended. Should be the user identifier (e.g. user_id). Used to maintain ordering within the Kafka stream — all messages with the same key are routed to the same partition, ensuring profile updates are applied in the correct sequence.
• Message headers: None
• Message body: JSON encoded item

Schema

{
  "user_id": "some_user",
  "user_attributes": {
    "any_attr": "any_value"
  },
  "timestamp": "2025-01-01 12:00:00"
}

Example

{
  "user_id": "3e685367-07d5-4d48-93ae-f007ac336605",
  "customer_id": 12345,
  "user_attributes": {
    "customer_tier": "VIP",
    "email": "[email protected]"
  },
  "timestamp": "2024-09-01 12:00:00.123"
}

High-volume domains — aggregation strategy

Where raw event volume is extremely high — such as spin-level gaming or rapid wallet deltas — we recommend aggregating data upstream before publishing to Kafka. This keeps throughput predictable, preserves business value, and helps manage event overage costs.

Session-based aggregates

Capped by time (e.g. ≤10 minutes) or count (e.g. ≤N spins). Each aggregate should include totals for stakes, wins, net, counts, game code, channel, and start/end timestamps.

Rolling window aggregates

Typically 2–5 minute summaries grouped by user or product.

Frequently asked questions

Related links

• External API Overview
• Attributes and Tags
• Real-Time Events