Archive for September, 2024

Taming the Kafka topics Wild West

Tuesday, September 17th, 2024

aka Approaches to managing Kafka topic creation with IBM Event Streams

How can you best operate central Kafka clusters, that can be shared by multiple different development teams?

Administrators talk about wanting to enable teams to create Kafka topics when they need them, but worry about it resulting in their Kafka clusters turning into a sprawling “Wild West”. At best, they talk about the mess of anonymous topics that are named and configured inconsistently. At worst, they talk about topics being created or configured in ways that negatively affect their Kafka cluster and impact their other users.

With that in mind, I wanted to share a few ideas for how to control the topics that are created in your Event Streams cluster:

(more…)

Flink can recognize when you’re cheating

Friday, September 13th, 2024

aka An unnecessarily complex and silly demo of MATCH_RECOGNIZE

I play a lot of video games. That includes a lot of modern games, but I also still love going back to the retro games of my childhood. There are a lot of fun things from that era of video games that I love.

For example, cheat codes. You’d press a specific sequence of buttons on the game controller at a specific time to unlock some “secret” bit of content – like special abilities, special resources, or levels.

Some of these are so ingrained in me now that my fingers just know how to enter them without thinking. The level select cheat for Sonic the Hedgehog is the best example of this: press UP, DOWN, LEFT, RIGHT, START + A during the title screen to access a level select mode that would let you jump immediately to any part of the game.


level select cheat code for Sonic the Hedgehog

With this in the back of my head, it’s perhaps no surprise that when I needed to explain pattern recognition in Apache Flink, the metaphor I thought of first was how games of yesteryear could recognize certain button press sequences.

If you think of each button press on the game controller as an event, then recognizing a cheat code is just a pattern of events to recognize.

And once I thought of the metaphor – I had to build it. 🙂

Version 1 (virtual controllers)

architecture diagram for the demo

There is more detail on how I built this in the git repository, but this is the overall idea for what I’ve made.

(more…)

Using quotas with Event Endpoint Management

Thursday, September 12th, 2024

In this post, we share examples of using quotas with IBM Event Endpoint Management, give you some pointers to help you try them for yourself, and most importantly get you thinking about where this might be useful for your own catalog.

Event Endpoint Management makes it easy for you to share your Kafka topics. Put some of your Kafka topics in the catalog, and allow colleagues and partners to discover the topics, so they can use the self-service catalog page to get started with them immediately.

Increasing reuse of your streams of events makes it possible for your business to unlock even more value from them. Innovative new ways to use them, that you might not have even thought of, will be enabled the more widely you share.

But before you invite colleagues and partners to start using your topics, you want to make sure that you’re ready. Event Endpoint Management offers a range of tools to make sure that you remain in control. Quotas are just one of these, and we dig into what they offer in this post.

Co-authored with Chris Patmore

(more…)

Using IBM Event Endpoint Management with Kafka Connect

Wednesday, September 11th, 2024

You’ve discovered a topic in the IBM Event Endpoint Management catalog that someone in your company has shared. It looks useful, so you want to use that stream of events to maintain a local projection in your database.

Or maybe you’ve discovered a topic in the Catalog that is available to produce to, and you want to contribute events to it from your MQ queue.

What are the options for using Kafka Connect to produce to, or consume from, topics that you discover in Event Endpoint Management?

In this post, we’ll share options that you can consider, and briefly outline the pros and cons of each.

Co-written with Andrew Borley

(more…)