I’ve made a Kafka Connect source connector for sending “real-time” events from Xbox LIVE to Kafka topics.
Quick primer if you’re not a gamer or don’t know what I’m talking about!
The Xbox platform comes with a social aspect: details about games you play and the achievements you earn playing them, are shared with your friends on the Xbox LIVE service. That is the source of data I’m using here.
Create an API key for your Xbox LIVE account, and run the Connector with it, and it will start producing two streams of events to your Kafka cluster:
Screenshot from Event Streams, but as a Kafka Connect connector, you could use any flavour of Kafka, including Apache Kafka.
ACHIEVEMENTSEvents when one of your friends earns an achievement.
PRESENCE
Events when your friends start playing a game, or go online/offline.
Details about the attributes of each of these events can be found in the Connector README, but here are a few screenshots to give you an idea.
An example message on the ACHIEVEMENTS topic.
Events look like this:
{
"date": "2022-11-30T22:13:31.78Z",
"gamertag": "nicthemighty",
"gamername": "Nic Townsend",
"name": "Beyond The Horizon",
"description": "Completed 'Radio Silence.'",
"icon": "http://images-eds.xboxlive.com/image?url=27S1DHqE.cHkmFg4nspsd6NoGj7jZQMki04gpiRpEPuwG3HCQIFJuvdOFdGo12yCzLSqj44HuZfLxpyO5k2.SV6ALPJOEvOsHra7GYt_BP0JRVg_FGOI2vo8IozeWvtHGXU01cfVQ0CXtbgDmatXRg--",
"contentname": "DEATHLOOP",
"contentimage": "http://store-images.s-microsoft.com/image/apps.5858.14634955238674857.649b7ff9-0dfc-4951-9b65-c5d815215da6.90208516-ba3b-47a9-a130-ef94cf860f5b",
"platform": "XboxOne",
"gamerscore": 30,
"rarityscore": 5,
"raritycategory": "Rare"
}
Notice that the Connector lets you specify a prefix to add to the topic names that it uses, which is how you can produce events to a topic like XBOX.LIVE.ACHIEVEMENTS like I am here.
Events on the PRESENCE topic will identify when a gamer goes online or offline. If the user is playing a game, the event will include the details of the game that they are playing.
Events look like this:
{
"date": "2022-12-19T01:52:03.961465802Z",
"userid": "2533274819921149",
"state": "Online",
"titleid": "2001700854",
"titlename": "Call of Duty®: Modern Warfare® II"
}
How does it work?
The Connector is just a quick-and-simple wrapper around the REST API from OpenXBL who provide an unofficial API for getting data from Xbox LIVE. It polls the APIs that they provide, and when it sees something new, emits that as a Kafka event.
The poll interval is configurable, but you can set it to poll every fifty seconds and still keep under the rate limit for a free tier API key.
Why did I make it?
I need a real-but-fun live source of events for a simple stream processing demo. I figured I might be able to do something fun with this.
For example, I can show enriching events, such as by making a stream processor that takes the PRESENCE events and looks up user details for the user id in the event.
And I can show examples of more stateful stream processing, such as emitting events when more than a few people are playing the same game at the same time, or emitting events when someone has been playing a game for longer than an hour.
I could’ve done this with a fake synthetic data source, but I thought it might be more interesting to do it with something real.
Want to use it?
This is just a quick hack for my own use, but I thought I’d share it. If you need (nearly) real-time Xbox events in kafka, you can find the Connector on github.
I’ve included documentation for the config parameters, and sample config parameters, in the repo as well.
Tags: apachekafka, kafka, xbox, xbox live