This is a talk I gave at Kafka Summit last month. It was an introduction to the Java APIs for Kafka Connect and Kafka Streams, using data from Xbox to bring the examples to life.
Confluent require personal details to watch recordings from Kafka Summit – sorry
It’s easy for developers who aren’t immersed in all-things-Kafka to assume that “Apache Kafka” just means an event backbone: something that hosts topics (and perhaps the client libraries to produce and consume messages using those topics). But Kafka is more than that. It is an ecosystem of tools that enables a complete event-streaming application.
That was the premise of this talk, recorded at Devoxx UK, which I gave to a room of Java developers. I introduced them to two other bits of Kafka: Kafka Connect (for getting data in and out of Kafka topics from external systems) and Kafka Streams (for developing stream processing applications).
Because they were Java developers, I thought the best way to give them a flavour of these tools was to show them the APIs, and walk through an example solution made using the APIs.
The example solution used Kafka tools to process data from Xbox – mostly because I’m a gamer and it made for a fun, if silly, demo.
I’ve made a Kafka Connect source connector for sending “real-time” events from Xbox LIVE to Kafka topics.
Quick primer if you’re not a gamer or don’t know what I’m talking about!
The Xbox platform comes with a social aspect: details about games you play and the achievements you earn playing them, are shared with your friends on the Xbox LIVE service. That is the source of data I’m using here.
Create an API key for your Xbox LIVE account, and run the Connector with it, and it will start producing two streams of events to your Kafka cluster:
Screenshot from Event Streams, but as a Kafka Connect connector, you could use any flavour of Kafka, including Apache Kafka.
ACHIEVEMENTSPRESENCE
Events when your friends start playing a game, or go online/offline.
Details about the attributes of each of these events can be found in the Connector README, but here are a few screenshots to give you an idea.
An example message on the ACHIEVEMENTS topic.
Happy Birthday to me, Happy Birthday to me…
A bunch of social networking updates arrived tonight for the Xbox 360. There has been talk of twitter integration in the Xbox 360 since E3 last summer. And after all the wait… I was a little disappointed.
Because it isn’t integration.
It’s a simple stand-alone twitter client app, that you can run on the Xbox. It joins the main menu becoming something else you can run, instead of play a game or media.
That’s not integration.
If they’d added the ability to receive replies/mentions and direct messages through the Xbox messaging platform while you’re in a game (as it does already with MSN Messenger messages)… that would be integrated.
If they’d built it in to Xbox’s Achievements system, so that when you complete an achievement it offered the chance to let your followers know what you’ve achieved… say by popping up a text box prefilled with a message and a link to more info about the game you’re playing… that would be integrated.
If they’d added the ability to capture a screenshot of what you’re doing, upload it to twitpic (or some similar service) and tweet to show your followers something cool… that would be integrated.
If they’d added the chance to see compare your twitter friends list with your Xbox friends list, and add any that aren’t on both… that would be integrated.
If they’d included a twitterfone-type service to let you tweet without typing, just by talking into the Xbox headset… that would be integrated.
I’m not saying that any of these are good ideas (they’re just the first few things off the top of my head), but the point is that this sort of thing would make twitter feel like it has been integrated into the “Xbox Experience“.
I’ve twittered this already, but for those people not following the inane minutiae of my life, here is another call.
I got an XBox 360 for my birthday (woot!) and am really getting into XBox Live games. If you know me and play on XBox Live, feel free to add me as a friend. My gamertag is dalelane.
And don’t be surprised if I become massively unproductive for the next few weeks…