This is a talk I gave at Kafka Summit last month. It was an introduction to the Java APIs for Kafka Connect and Kafka Streams, using data from Xbox to bring the examples to life.
Confluent require personal details to watch recordings from Kafka Summit – sorry
It’s easy for developers who aren’t immersed in all-things-Kafka to assume that “Apache Kafka” just means an event backbone: something that hosts topics (and perhaps the client libraries to produce and consume messages using those topics). But Kafka is more than that. It is an ecosystem of tools that enables a complete event-streaming application.
That was the premise of this talk, recorded at Devoxx UK, which I gave to a room of Java developers. I introduced them to two other bits of Kafka: Kafka Connect (for getting data in and out of Kafka topics from external systems) and Kafka Streams (for developing stream processing applications).
Because they were Java developers, I thought the best way to give them a flavour of these tools was to show them the APIs, and walk through an example solution made using the APIs.
The example solution used Kafka tools to process data from Xbox – mostly because I’m a gamer and it made for a fun, if silly, demo.
I’ve made a Kafka Connect source connector for sending “real-time” events from Xbox LIVE to Kafka topics.
Quick primer if you’re not a gamer or don’t know what I’m talking about!
The Xbox platform comes with a social aspect: details about games you play and the achievements you earn playing them, are shared with your friends on the Xbox LIVE service. That is the source of data I’m using here.
Create an API key for your Xbox LIVE account, and run the Connector with it, and it will start producing two streams of events to your Kafka cluster:
Screenshot from Event Streams, but as a Kafka Connect connector, you could use any flavour of Kafka, including Apache Kafka.
ACHIEVEMENTSPRESENCE
Events when your friends start playing a game, or go online/offline.
Details about the attributes of each of these events can be found in the Connector README, but here are a few screenshots to give you an idea.
An example message on the ACHIEVEMENTS topic.
I’ve twittered this already, but for those people not following the inane minutiae of my life, here is another call.
I got an XBox 360 for my birthday (woot!) and am really getting into XBox Live games. If you know me and play on XBox Live, feel free to add me as a friend. My gamertag is dalelane.
And don’t be surprised if I become massively unproductive for the next few weeks…