strimzi

Posts Tagged ‘strimzi’

Using annotations to store info about Kafka topics in Strimzi

Sunday, June 1st, 2025

In this post, I highlight the benefits of using Kubernetes annotations to store information about Kafka topics, and share a simplified example of how this can even be automated.

Managing Kafka topics as Kubernetes resources brings many benefits. For example, they enable automated creation and management of topics as part of broader CI/CD workflows, it gives a way to track history of changes to topics and avoid configuration drift as part of GitOps processes, and they give a point of control for enforcing policies and standards.

The value of annotations

Another benefit that I’ve been seeing increasing interest in recently is that they provide a cheap and simple place to store small amounts of metadata about topics.

For example, you could add annotations to topics that identify the owning application or team.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: some-kafka-topic
  annotations:
    acme.com/topic-owner: 'Joe Bloggs'
    acme.com/topic-team: 'Finance'

Annotations are simple key/value pairs, so you can add anything that might be useful to a Kafka administrator.

You can add links to team documentation.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: some-kafka-topic
  annotations:
    acme.com/documentation: 'https://acme-intranet.com/finance-apps/some-kafka-app'

You can add a link to the best Slack channel to use to ask questions about the topic.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: some-kafka-topic
  annotations:
    acme.com/slack: 'https://acme.enterprise.slack.com/archives/C2QSX23GH'

(more…)

Tags:apachekafka, ibmeventstreams, kafka, strimzi
Posted in code, ibm | No Comments »

Migrating your Apache Kafka cluster using MirrorMaker 2

Wednesday, March 24th, 2021

You have a Kafka cluster that you have been using for a while. Your cluster has many topics, and the topics have many messages.

Now you’ve decided to move and start using a new, different Kafka cluster somewhere else.

How can you take your topics with you?

Huge thanks to Andrew Borley for co-writing this with me. Useful insights in here probably came from him, the mistakes from me.

(more…)

Tags:apachekafka, kafka, strimzi
Posted in code | Comments Closed

Using TensorFlow to make predictions from Kafka events

Sunday, September 6th, 2020

This post is a simple example of how to use a machine learning model to make predictions on a stream of events on a Kafka topic.

It’s more a quick hack than a polished project, with most of this code hacked together from samples and starter code in a single evening. But it’s a fun demo, and could be a jumping-off point for starting a more serious project.

For the purposes of a demo, I wanted to make a simple example of how to implement this pattern, using:

sensors that are easily and readily available, and
predictions that are easy to understand (and easy to generate labelled training data for)

With that goal in mind, I went with:

for the sensors providing the source of events, I used the accelerometer and gyroscope on my iPhone
to set up the Kafka broker, I used the Strimzi Kafka Operator
for the machine learning model, I used TensorFlow to make a simple bidirectional LSTM
the predictions I’m making are a description of what I’m doing with the phone (e.g. is it in my hand, is it in my pocket, etc.)

I’ve got my phone publishing a live stream of raw sensor readings, and passing that stream through an ML model to give me a live stream of events like “phone has been put on a table”, “phone has been picked up and is in my hand”, or “phone has been put in a pocket while I’m sat down”, etc.

Here is it in action. It’s a bit fiddly to demo, and a little awkward to film putting something in your pocket without filming your lap, so bear with me!