Archive for the ‘code’ Category

Getting demo events onto IBM Event Streams topics in a hurry

Wednesday, March 13th, 2024

Sharing a couple of tips for quick-and-dirty demo setups.

I often need to put together demos of IBM Event Automation without much notice. The starting point is almost always needing to get a bunch of interesting events onto a Kafka topic.

What I need is a jumping-off point to illustrate the benefit of sharing streams of events in Event Endpoint Management, or the types of processing you can do in Event Processing. And to do that, I need a topic with events on them that will look interesting or relevant to who I’m demo’ing to.

If I’ve got time to do this properly, I’ll setup a generator that will give me a continuous stream of randonly-generated events (example). But if I’m in a hurry, I’ll use the REST Producer API and do something like this instead.

(more…)

Explaining regression in Scratch

Thursday, February 15th, 2024

In this post, I want to share a preview of a new feature I’m adding to Machine Learning for Kids, to ask for feedback and ideas for projects that it could be used to make.

I’ll start by contrasting this new feature with what I’ve done with Machine Learning for Kids before, then I’ll share screenshots of the new feature, and finally I’ve got a ten-minute video showing the sort of school lesson that I think it could be used for.

(more…)

How to create a new Kafka Connect connector

Sunday, January 28th, 2024

I’m helping with a hackathon this week to get developers to create their first Kafka Connect connectors. To help get them going, I’ll be starting the day with a quick crash course.

I’ve created a simple skeleton connector project, and I’ll be walking through how they can use that to skip the boilerplate stuff and jump straight into trying out their ideas for new connectors.

(more…)

Processing XML with Kafka Connect

Monday, December 11th, 2023

In this tutorial, I’ll share examples of how to process XML data at various points in a Kafka Connect pipeline, using a new plugin from IBM Event Streams.

You can assemble a Kafka Connect pipeline in a huge number of ways, so I’m not going to attempt an exhaustive list here. Instead, I’ve come up with eight examples that are illustrative of the sort of use cases you can satisfy.

I’ll summarise and link to my different examples here, so you can jump straight to the one that sounds the closest to your use case:

(more…)

You need two schemas to deserialize an Avro message… but which two?

Friday, November 17th, 2023

In this post, I want to talk about what happens when you use Avro to deserialize messages on a Kafka topic, why it actually needs two schemas, and what those schemas need to be.

I should start by pointing out that if you’re using a schema registry, you probably don’t need to worry about any of this. In fact, a TLDR for this whole post could be “You should be using a good schema registry and SerDes client“.

But, there are times where this may be difficult to do, so knowing how to set a deserializer up correctly is helpful. (Even if you’re doing the right thing and using a Schema Registry, it is still interesting to poke at some of the details and know what is happening.)

The key thing to understand is that to deserialize binary-encoded Avro data, you need a copy of the schema that was used to serialize the data in the first place [1].


This gets interesting after your topic has been around for a while, and you have messages using a mixture of schema versions on the topic. Maybe over the lifetime of your app, you’ve needed to add new fields to your messages a couple of times.

If you want a consumer application to be able to consume all of the messages on this topic, what does that mean?

(more…)

Machine Learning for Kids with EduBlocks

Saturday, July 8th, 2023

Students can now create Machine Learning for Kids projects using EduBlocks – letting them create machine learning Python projects in the browser by dragging and dropping blocks on a canvas.

This is all thanks to a fantastic new contribution from Joshua Lowe.

Here’s a quick run-through to show what this makes possible.

(more…)

Using Xbox to get started with Kafka Connect & Kafka Streams

Wednesday, May 17th, 2023

It’s easy for developers who aren’t immersed in all-things-Kafka to assume that “Apache Kafka” just means an event backbone: something that hosts topics (and perhaps the client libraries to produce and consume messages using those topics). But Kafka is more than that. It is an ecosystem of tools that enables a complete event-streaming application.

That was the premise of this talk, recorded at Devoxx UK, which I gave to a room of Java developers. I introduced them to two other bits of Kafka: Kafka Connect (for getting data in and out of Kafka topics from external systems) and Kafka Streams (for developing stream processing applications).

Because they were Java developers, I thought the best way to give them a flavour of these tools was to show them the APIs, and walk through an example solution made using the APIs.

The example solution used Kafka tools to process data from Xbox – mostly because I’m a gamer and it made for a fun, if silly, demo.


recording of the talk at youtu.be/rvVjI1NXswU

(more…)

Using Apache Kafka with IBM MQ using Kafka Connect

Thursday, April 20th, 2023

A recording of a demo walkthrough I did about using the Kafka Connect MQ connectors to flow messages between IBM MQ and Apache Kafka.

A few weeks ago, I presented a session at TechCon about IBM MQ and Apache Kafka with David Ware. I spent most of my time running through how to use Kafka Connect with IBM MQ, with a few demos showing different ways to setup and run the kafka-connect-mq-source Connector.

My demos start at around 20 minutes in, but you should listen to David give the context first!