Archive for October, 2019

Using TensorFlow with IBM Event Streams
(Kafka + Machine Learning = Awesome)

Thursday, October 31st, 2019

In this post, I want to explain how to get started creating machine learning applications using the data you have on Kafka topics.

I’ve written a sample app, with examples of how you can use Kafka topics as:

  • a source of training data for creating machine learning models
  • a source of test data for evaluating machine learning models
  • an ongoing stream of events to make predictions about using machine learning models

I’ll use this post to explain how it works, and how you can use it as the basis of writing your first ML pipeline using the data on your own Kafka topics.


NASA Space Apps Challenge at Hursley

Sunday, October 20th, 2019

This weekend was NASA Space Apps Challenge again – a weekend space-themed hackathon organised by NASA. It runs around the world, and this year IBM Hursley hosted one again.

I was in a small team with Faith. There were a variety of challenges to choose from and we chose Orbital Scrap Metal which was about educating the public about orbital debris, or space junk – explaining what it is, where it comes from, and the potential impact it has.

We created a game to help kids learn about space debris while playing. It’s fun, educational, and is all driven by real live data about space debris – each time you play, you interact with different real debris items.


Using Avro schemas from Python apps with IBM Event Streams

Thursday, October 17th, 2019

I’ve written before about how to write a schema for your developers using Kafka. The examples I used before were all in Java, but someone asked me yesterday if I could share some Python equivalents.

The principles are described in the Event Streams documentation, but in short, your Kafka producers use Apache Avro to serialize the message data that you send, and identify the schema that you’ve used in the Kafka message header. In your Kafka consumers, you look at the headers of the messages that you receive to know which schema to retrieve, and use that to deserialize message data.