Archive for the ‘ibm’ Category

Taming the Kafka topics Wild West

Tuesday, September 17th, 2024

aka Approaches to managing Kafka topic creation with IBM Event Streams

How can you best operate central Kafka clusters, that can be shared by multiple different development teams?

Administrators talk about wanting to enable teams to create Kafka topics when they need them, but worry about it resulting in their Kafka clusters turning into a sprawling “Wild West”. At best, they talk about the mess of anonymous topics that are named and configured inconsistently. At worst, they talk about topics being created or configured in ways that negatively affect their Kafka cluster and impact their other users.

With that in mind, I wanted to share a few ideas for how to control the topics that are created in your Event Streams cluster:

(more…)

Using quotas with Event Endpoint Management

Thursday, September 12th, 2024

In this post, we share examples of using quotas with IBM Event Endpoint Management, give you some pointers to help you try them for yourself, and most importantly get you thinking about where this might be useful for your own catalog.

Event Endpoint Management makes it easy for you to share your Kafka topics. Put some of your Kafka topics in the catalog, and allow colleagues and partners to discover the topics, so they can use the self-service catalog page to get started with them immediately.

Increasing reuse of your streams of events makes it possible for your business to unlock even more value from them. Innovative new ways to use them, that you might not have even thought of, will be enabled the more widely you share.

But before you invite colleagues and partners to start using your topics, you want to make sure that you’re ready. Event Endpoint Management offers a range of tools to make sure that you remain in control. Quotas are just one of these, and we dig into what they offer in this post.

Co-authored with Chris Patmore

(more…)

Using IBM Event Endpoint Management with Kafka Connect

Wednesday, September 11th, 2024

You’ve discovered a topic in the IBM Event Endpoint Management catalog that someone in your company has shared. It looks useful, so you want to use that stream of events to maintain a local projection in your database.

Or maybe you’ve discovered a topic in the Catalog that is available to produce to, and you want to contribute events to it from your MQ queue.

What are the options for using Kafka Connect to produce to, or consume from, topics that you discover in Event Endpoint Management?

In this post, we’ll share options that you can consider, and briefly outline the pros and cons of each.

Co-written with Andrew Borley

(more…)

Unleash real-time responsiveness by enriching streams of events

Wednesday, June 19th, 2024

In this demo, I give examples of different ways to enrich a stream of events – and the types of event processing that this can enable.

I presented this in a webinar with Matt Sunley (replay recording available) yesterday. Matt started with some context, explaining how enriching events with data from external sources can enable event processing solutions that aren’t otherwise possible.

And then I ran through a demo, creating an event processing flow that included four types of enrichment.

(more…)

Using IBM Event Automation with Amazon MSK

Wednesday, October 25th, 2023

Written with Chris Patmore

IBM Event Automation helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have.

A key goal of Event Automation is to be composable. The three components can be used together, or they can each be used to extend and enhance an existing event-driven deployment.

Amazon MSK (Managed Streaming for Kafka) is a hosted, managed Kafka service available in Amazon Web Services. If a business has started their event-driven journey using MSK, then components from Event Automation can help to enhance this. This could be by offering management and governance of their MSK topics. And it could be by providing an intuitive low-code authoring canvas to process the events on their MSK topics.

Working with Amazon MSK is a nice example of the benefits of the composability of Event Automation, by helping businesses to get more value from their existing MSK topics.

In this blog post, we want to show a few different examples of where this can be done. For each example, we’ll provide a high-level diagram and description. We’ll also share a demonstration that we created to show it in action.

(more…)

Connecting App Connect Enterprise to Event Endpoint Management

Friday, October 20th, 2023

Configuring IBM App Connect Enterprise to consume messages from Kafka topics in IBM Event Endpoint Management requires careful configuration. In this post, I’ll share the steps I use that help me to avoid missing any required values.

If this sounds familiar, it might be because I wrote a post like this about using App Connect Enterprise to work with topics from Event Streams. People seem to have found that post useful, so I thought I’d do something similar for topics in Event Endpoint Management this time.

To illustrate this, I’ll create a simple App Connect flow that consumes messages from a Kafka topic and publish them to an MQTT topic.

The key to getting this to work correctly first time is to make sure that values are accurately copied from Event Endpoint Management to App Connect.

To help with this, I use a grid like the one below.

The instructions in this post start with Event Endpoint Management, and explain how to populate the grid with the information you need.

Then the instructions will switch to App Connect, and explain how to use the values in the grid to set up your App Connect flow.

What this is Values you will see in my screenshots Your value
A Topic name
DEMO.ACE
B Bootstrap address
my-eem-gateway-ibm-egw-rt-event-automation.apps.dalelane.cp.fyre.ibm.com:443
C SASL mechanism
PLAIN
D SASL config
org.apache.kafka.common.security.plain.PlainLoginModule required;
E Security protocol
SASL_SSL
F Certificate
eem-cert.jks
G Certificate password
STOREPASSW0RD
H Username
eem-9c8fc5d9-fddd-48dd-ab41-e062214166e5
I Password
dd08a1fc-99be-4931-8059-70aef88c1f0c
J Policy project name
demo-policies
K Policy name
demo-eem-policy
L Security identity name
eem-credentials
M Truststore identity name
eem-truststore

(more…)

Understanding windows in Event Processing

Wednesday, October 11th, 2023

Event Processing (one of the new capabilities of IBM Event Automation) makes it easy to perform stateful processing of streams of events from Kafka topics. In this post, I want to drill down a little into how windowed processing behaves.

I’ve enjoyed the chance to see the solutions that people have started to build with the tool. As part of this, I’ve been helping several people to understand the results produced by the event processing flows they’ve made.

These often started with different questions, such as:

  • Why hasn’t my flow produced any results?
  • Why isn’t my one-minute window producing one result every minute?
  • Why did the last one event on my Kafka topic cause results to be produced for several different windows?
  • etc.

However, these are often symptoms of a single common question: how windowed processing operations behave.

I’ve tried to come up with simple ways to demonstrate how it works, so in this post I want to share how I’m currently explaining it.


youtu.be/x_r6GNZmsd4

What can you do with IBM Event Automation?

Sunday, July 9th, 2023

This is IBM Event Automation : a new product we released last month to help our clients create event driven solutions.

I’ve written a 200-word summary of what IBM Event Automation is, but in this post I wanted to dive a little bit deeper and show what it can do.

(more…)