Archive for the ‘ibm’ Category

Running OpenMessaging benchmarks on your Kafka cluster

Monday, February 3rd, 2025

The OpenMessaging Benchmark Framework is typically used to benchmark messaging systems in the cloud, but in this post I want to show how useful it can also be for Kafka clusters that you run yourself in Kubernetes (whether that is using the open source Strimzi operator, or IBM’s Event Streams).

From openmessaging.cloud:

The OpenMessaging Benchmark Framework is a suite of tools that make it easy to benchmark distributed messaging systems.

As I’ve written about before (when illustrating the impact of setting quotas at the Kafka cluster level, and when adding quotas at the event gateway level), Apache Kafka comes with a good performance test tool. That is still my go-to option if I just want an easy way to push data through a Kafka cluster in bulk.

But – OpenMessaging’s benchmark has some interesting features that make it a useful complement and worth considering.

The benefit that OpenMessaging talk about the most is that can be used with a variety of messaging systems, such as RocketMQ, Pulsar, RabbitMQ, NATS, Redis and more – although in this post I’m only interested in using it to benchmark an Apache Kafka cluster.

More interesting for me was their focus on realistic workloads rather than relying on static data.

Quoting again from openmessaging.cloud:

Benchmarks should be largely oriented toward standard use cases rather than bizarre edge cases

(more…)

API Enrichment in Event Processing

Monday, December 2nd, 2024

Creating an Apache Flink job using IBM Event Processing that identifies customer retention opportunities

This is a demo of IBM Event Processing that I gave today. The focus was meant to be how API Enrichment (enriching a stream of events with reference data from external sources) can inform the processing of your Flink job, but it ended up being a useful example of filtering and transforming, too.


narrated demo at youtu.be/mzlQZBVg6HA

(more…)

Using IBM Event Processing with rules engines

Tuesday, November 12th, 2024

In this post, I’ll demonstrate how Event Processing can use parameters from an external source (such as a rules engine) in event processing flows.

A simple flow to demonstrate the idea

To illustrate the idea, I created a simple demo event processing flow. The flow takes a stream of order events, filters it to keep only orders for high value items, and then modifies the description property in some of the events:

The filter node is comparing the price with “40”, so only order events for items with a value above $40 are kept.

The transform node is modifying the description property of order events – any description that contains the string “Cargo Jeans” is replaced with “Combat Trousers”.

Hard-coded parameters

What if you wanted to modify the threshold for the filter, to change that $40 minimum value for an order to be considered “large”?

Or what if you wanted to modify the transformation, so that different strings would be used in the regular expression replacement?

With the values hard-coded in the flow as shown above, you would need to:

  • create a savepoint for the job
  • stop the job
  • modify the parameters in the job
  • resume the job from the savepoint

This is a workable approach, although it does require a little downtime and some administrative effort.

The aim for this post is to highlight an alternative approach.

(more…)

Event-driven tech at IBM TechXchange

Saturday, October 19th, 2024

This week, I’m at IBM TechXchange: our annual technical learning conference.

Our other big annual event Think has a business focus, but TechXchange is for technologists to advance their skills and expertise.

There are thousands of presentations, demos, workshops and hands-on labs to choose from, but naturally the most interesting ones will be about event-driven architectures and event stream processing technologies. 😉

In this post, I’ll share a few of our sessions from each day – if you’re at TechXchange this week, I hope to see you at some of these!

(more…)

Analysing Wikipedia edits with IBM Event Processing

Monday, October 14th, 2024

In this post, I’ll share a demo I gave today to explain some of the processing nodes in the palette of IBM Event Processing.

I’ve found that demonstrations of Event Processing are easier to understand when I don’t need to explain the stream of events I’m processing in the first place. This means I’m always looking for interesting real-world event streams that are widely understood, as they can make for the most effective demos.

With this in mind, today I tried explaining a few of the Event Processing nodes by using them with a live stream of events representing pages that are being created and edited in the English Wikipedia.


Click on the image for a higher-resolution screenshot

Each event contains:

  • title of the page
  • who made the edit (user ID if logged in, or IP address if anonymous)
  • was this the creation of a new page, or an edit of an existing page?

Every edit on Wikipedia results in an event on the Kafka topic, so there are typically a few events a second. It’s not a super-high-throughput topic in Kafka terms, but there are enough events to try out interesting ideas.


Click on the image for a higher-resolution screenshot

Here are a few of the demos I gave today.

This is by no means an exhaustive list of what you could do with this data, but it was enough to let me show what the most commonly-used tools in the palette can do.

(more…)

Taming the Kafka topics Wild West

Tuesday, September 17th, 2024

aka Approaches to managing Kafka topic creation with IBM Event Streams

How can you best operate central Kafka clusters, that can be shared by multiple different development teams?

Administrators talk about wanting to enable teams to create Kafka topics when they need them, but worry about it resulting in their Kafka clusters turning into a sprawling “Wild West”. At best, they talk about the mess of anonymous topics that are named and configured inconsistently. At worst, they talk about topics being created or configured in ways that negatively affect their Kafka cluster and impact their other users.

With that in mind, I wanted to share a few ideas for how to control the topics that are created in your Event Streams cluster:

(more…)

Using quotas with Event Endpoint Management

Thursday, September 12th, 2024

In this post, we share examples of using quotas with IBM Event Endpoint Management, give you some pointers to help you try them for yourself, and most importantly get you thinking about where this might be useful for your own catalog.

Event Endpoint Management makes it easy for you to share your Kafka topics. Put some of your Kafka topics in the catalog, and allow colleagues and partners to discover the topics, so they can use the self-service catalog page to get started with them immediately.

Increasing reuse of your streams of events makes it possible for your business to unlock even more value from them. Innovative new ways to use them, that you might not have even thought of, will be enabled the more widely you share.

But before you invite colleagues and partners to start using your topics, you want to make sure that you’re ready. Event Endpoint Management offers a range of tools to make sure that you remain in control. Quotas are just one of these, and we dig into what they offer in this post.

Co-authored with Chris Patmore

(more…)

Using IBM Event Endpoint Management with Kafka Connect

Wednesday, September 11th, 2024

You’ve discovered a topic in the IBM Event Endpoint Management catalog that someone in your company has shared. It looks useful, so you want to use that stream of events to maintain a local projection in your database.

Or maybe you’ve discovered a topic in the Catalog that is available to produce to, and you want to contribute events to it from your MQ queue.

What are the options for using Kafka Connect to produce to, or consume from, topics that you discover in Event Endpoint Management?

In this post, we’ll share options that you can consider, and briefly outline the pros and cons of each.

Co-written with Andrew Borley

(more…)