You need two schemas to deserialize an Avro message… but which two?

November 17th, 2023

In this post, I want to talk about what happens when you use Avro to deserialize messages on a Kafka topic, why it actually needs two schemas, and what those schemas need to be.

I should start by pointing out that if you’re using a schema registry, you probably don’t need to worry about any of this. In fact, a TLDR for this whole post could be “You should be using a good schema registry and SerDes client“.

But, there are times where this may be difficult to do, so knowing how to set a deserializer up correctly is helpful. (Even if you’re doing the right thing and using a Schema Registry, it is still interesting to poke at some of the details and know what is happening.)

The key thing to understand is that to deserialize binary-encoded Avro data, you need a copy of the schema that was used to serialize the data in the first place [1].

This gets interesting after your topic has been around for a while, and you have messages using a mixture of schema versions on the topic. Maybe over the lifetime of your app, you’ve needed to add new fields to your messages a couple of times.

If you want a consumer application to be able to consume all of the messages on this topic, what does that mean?

Read the rest of this entry »

Using IBM Event Automation with Amazon MSK

October 25th, 2023

Written with Chris Patmore

IBM Event Automation helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have.

A key goal of Event Automation is to be composable. The three components can be used together, or they can each be used to extend and enhance an existing event-driven deployment.

Amazon MSK (Managed Streaming for Kafka) is a hosted, managed Kafka service available in Amazon Web Services. If a business has started their event-driven journey using MSK, then components from Event Automation can help to enhance this. This could be by offering management and governance of their MSK topics. And it could be by providing an intuitive low-code authoring canvas to process the events on their MSK topics.

Working with Amazon MSK is a nice example of the benefits of the composability of Event Automation, by helping businesses to get more value from their existing MSK topics.

In this blog post, we want to show a few different examples of where this can be done. For each example, we’ll provide a high-level diagram and description. We’ll also share a demonstration that we created to show it in action.

Read the rest of this entry »

Connecting App Connect Enterprise to Event Endpoint Management

October 20th, 2023

Configuring IBM App Connect Enterprise to consume messages from Kafka topics in IBM Event Endpoint Management requires careful configuration. In this post, I’ll share the steps I use that help me to avoid missing any required values.

If this sounds familiar, it might be because I wrote a post like this about using App Connect Enterprise to work with topics from Event Streams. People seem to have found that post useful, so I thought I’d do something similar for topics in Event Endpoint Management this time.

To illustrate this, I’ll create a simple App Connect flow that consumes messages from a Kafka topic and publish them to an MQTT topic.

The key to getting this to work correctly first time is to make sure that values are accurately copied from Event Endpoint Management to App Connect.

To help with this, I use a grid like the one below.

The instructions in this post start with Event Endpoint Management, and explain how to populate the grid with the information you need.

Then the instructions will switch to App Connect, and explain how to use the values in the grid to set up your App Connect flow.

What this is Values you will see in my screenshots Your value
A Topic name
B Bootstrap address
C SASL mechanism
D SASL config required;
E Security protocol
F Certificate
G Certificate password
H Username
I Password
J Policy project name
K Policy name
L Security identity name
M Truststore identity name

Read the rest of this entry »

Understanding windows in Event Processing

October 11th, 2023

Event Processing (one of the new capabilities of IBM Event Automation) makes it easy to perform stateful processing of streams of events from Kafka topics. In this post, I want to drill down a little into how windowed processing behaves.

I’ve enjoyed the chance to see the solutions that people have started to build with the tool. As part of this, I’ve been helping several people to understand the results produced by the event processing flows they’ve made.

These often started with different questions, such as:

  • Why hasn’t my flow produced any results?
  • Why isn’t my one-minute window producing one result every minute?
  • Why did the last one event on my Kafka topic cause results to be produced for several different windows?
  • etc.

However, these are often symptoms of a single common question: how windowed processing operations behave.

I’ve tried to come up with simple ways to demonstrate how it works, so in this post I want to share how I’m currently explaining it.

My office “pet”

September 8th, 2023

I have a first-floor office by a large window. On the outside of the window is a wide ledge. And a few weeks ago, a couple of pigeons started assembling a nest of twigs on the ledge by my desk.

Read the rest of this entry »

MQTT extension for Scratch

August 5th, 2023


Extensions in Scratch let you add additional blocks to the palette. I’ve written about how to create extensions before, but in this post I want to share my latest extension which adds MQTT support.

I don’t have a particular Scratch project in mind for this yet, but publishing and subscribing to an MQTT broker from a Scratch project would allow multiple web browsers each running Scratch to communicate with each other. I’m sure there are some fun things this could be used for.

Read the rest of this entry »

Trying new things

July 27th, 2023

I’m a fan of trying unusual things. In this post, I thought I’d share some examples of things I’ve done in the last year or so, in case it gives you an idea of something you might want to try.

Read the rest of this entry »

Catching the carpool cheats

July 24th, 2023

I’ve added a new project worksheet to Machine Learning for Kids, based on the idea of a traffic camera in a carpool lane.

Students create training examples by labelling pictures of cars with random passengers in.

recording of creating the training examples at

Read the rest of this entry »