Looking back at my career so far, and what this could mean for what comes next…
Archive for the ‘ibm’ Category
22 years at IBM
Sunday, April 26th, 2026Talking at TechCon about AI and EDA
Sunday, March 22nd, 2026IBM TechCon is an annual online technical event for engineers, creators, and integration specialists.
One of our sessions for this year was AI patterns in event-driven architectures:
You already have Kafka topics sending valuable event data through your systems. You’ve heard about the increasing adoption and promise of AI technologies. But how do these worlds overlap?
We’ll explain the ways that you can take your Kafka topics and use them not just for integration and analytics, but also to drive AI and ML — things like real-time anomaly detection, prediction models, personalization, decision pipelines, and more.
In this session, we’ll explain the four main patterns for how your existing Kafka topics (and the streams of events on them) can be leveraged as the foundation for AI/ML. We’ll show multiple technology approaches to implement each pattern – rather than convince you to use any single specific tool, help you understand the high level patterns, and how can you get started with them.
session recording on video.ibm.com
This was adapted from a talk I gave at Current last year.
I got to go to a fancy convention centre in New Orleans for that talk, but I gave this one from a poorly-lit meeting room in my office… so in that respect at least, this one was less fun! 😉
But it’s a topic I find super interesting, so I am always pleased to have another chance to share my thoughts.
Talking at TechCon about metrics and monitoring
Saturday, March 21st, 2026IBM TechCon is an annual online technical event for engineers, creators, and integration specialists.
One of our sessions for this year was an introduction to Monitoring your Event Driven Architecture:
This session will give you an insight into the life of an Event Automation administrator, responsible for a busy event-driven system where teams have been creating a variety of Kafka topics, integrations, stream processing apps, connectors, and much more. We’ll highlight the importance of metrics and monitoring for event driven architectures and introduce you to the tools that are available to help.
We’ll do this by showing you an event-driven environment where things have gotten out of hand. In our fictional scenario, users are being impacted by things like poorly configured topics, poorly written applications, poorly managed connectors, poorly configured stream processors…
In this session, we’ll walk you through to bring control to the chaos. We’ll step through how to get an insight into what is happening, find out where the problems are, and put controls in place to mitigate their impact.
session recording on video.ibm.com
It was an introduction for beginners, that you could sum up as a 40-minute plea for people to monitor their Kafka clusters and applications! Essentially, we set up a handful of naive and broken applications, and walked through how metrics and monitoring show you where the problems are hiding.
Watch it to be persuaded that metrics are important.
Or to watch how Matt had to jump in and help me when an Apple Magic Mouse decide it didn’t like scrolling any more, and I needed to get to things at the bottom of web pages!
Or just to marvel at how glamourous our offices are. 😉
Deploying Apache Flink jobs into Kubernetes
Friday, March 20th, 2026IBM TechCon is an annual online technical event for engineers, creators, and integration specialists.
One of our sessions for this year was Deploying an Apache Flink job into production:
You’ve maybe seen the low-code canvas in Event Processing or the simple expressiveness of Flink SQL, and how easy they make it to author event stream processing. A business user who understands the data in the event stream can easily describe the patterns they’re interested in or the insights they want to look for. But what comes next?
In this session, we’ll walk through the ops tasks involved in taking that event processing flow, and deploying it into Kubernetes as a Flink application ready for production.
We’ll outline the steps that are needed and describe the main decisions you need to make. This includes the sorts of values you will want to monitor to make sure that your Flink application continues to run correctly.
It was a live walk-through of the steps involved in deploying Flink jobs in Kubernetes. I used Event Processing to create the Flink job that I used for the demos, because low-code UI’s are easier to follow in a presentation, but most of what I showed is applicable however you’ve created your Flink job – and was a high-level introduction to using the Flink Kubernetes Operator.
Event-driven sessions at IBM TechXchange 2025
Tuesday, September 30th, 2025
Next week, I’ll be at IBM TechXchange: our annual technical learning conference.
Our other big annual event Think had a business focus, but TechXchange is for technologists to advance their skills and expertise.
There are thousands of presentations, demos, workshops and hands-on labs to choose from, but naturally the most interesting ones will be about event-driven architectures and event stream processing technologies. 😉
In this post, I’ll share what I’ll be up to – if you’re going to TechXchange next week, I hope to see you at some of these!
Using annotations to store info about Kafka topics in Strimzi
Sunday, June 1st, 2025In this post, I highlight the benefits of using Kubernetes annotations to store information about Kafka topics, and share a simplified example of how this can even be automated.
Managing Kafka topics as Kubernetes resources brings many benefits. For example, they enable automated creation and management of topics as part of broader CI/CD workflows, it gives a way to track history of changes to topics and avoid configuration drift as part of GitOps processes, and they give a point of control for enforcing policies and standards.
The value of annotations
Another benefit that I’ve been seeing increasing interest in recently is that they provide a cheap and simple place to store small amounts of metadata about topics.
For example, you could add annotations to topics that identify the owning application or team.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: some-kafka-topic
annotations:
acme.com/topic-owner: 'Joe Bloggs'
acme.com/topic-team: 'Finance'
Annotations are simple key/value pairs, so you can add anything that might be useful to a Kafka administrator.
You can add links to team documentation.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: some-kafka-topic
annotations:
acme.com/documentation: 'https://acme-intranet.com/finance-apps/some-kafka-app'
You can add a link to the best Slack channel to use to ask questions about the topic.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: some-kafka-topic
annotations:
acme.com/slack: 'https://acme.enterprise.slack.com/archives/C2QSX23GH'
Running OpenMessaging benchmarks on your Kafka cluster
Monday, February 3rd, 2025The OpenMessaging Benchmark Framework is typically used to benchmark messaging systems in the cloud, but in this post I want to show how useful it can also be for Kafka clusters that you run yourself in Kubernetes (whether that is using the open source Strimzi operator, or IBM’s Event Streams).
From openmessaging.cloud:
The OpenMessaging Benchmark Framework is a suite of tools that make it easy to benchmark distributed messaging systems.
As I’ve written about before (when illustrating the impact of setting quotas at the Kafka cluster level, and when adding quotas at the event gateway level), Apache Kafka comes with a good performance test tool. That is still my go-to option if I just want an easy way to push data through a Kafka cluster in bulk.
But – OpenMessaging’s benchmark has some interesting features that make it a useful complement and worth considering.
The benefit that OpenMessaging talk about the most is that can be used with a variety of messaging systems, such as RocketMQ, Pulsar, RabbitMQ, NATS, Redis and more – although in this post I’m only interested in using it to benchmark an Apache Kafka cluster.
More interesting for me was their focus on realistic workloads rather than relying on static data.
Quoting again from openmessaging.cloud:
Benchmarks should be largely oriented toward standard use cases rather than bizarre edge cases
API Enrichment in Event Processing
Monday, December 2nd, 2024Creating an Apache Flink job using IBM Event Processing that identifies customer retention opportunities
This is a demo of IBM Event Processing that I gave today. The focus was meant to be how API Enrichment (enriching a stream of events with reference data from external sources) can inform the processing of your Flink job, but it ended up being a useful example of filtering and transforming, too.