Posts Tagged ‘ibmeventstreams’

Using Mirror Maker 2 with IBM Event Streams to broadcast events to multiple regions

Tuesday, April 2nd, 2024

This is the second in a series of blog posts sharing examples of ways to use Mirror Maker 2 with IBM Event Streams.

Mirror Maker 2 is a powerful and flexible tool for moving Kafka events between Kafka clusters, but sometimes I feel like this can be forgotten if we only talk about it in the context of disaster recovery.

In these posts, I want to inspire you to think about other ways you could use Mirror Maker 2. The best way to learn about what is possible is to play with it for yourself, so with these posts I’ll include a script to create a demonstration of the scenario.

For this second post, I’ll look at using Mirror Maker to broadcast events to clusters in multiple regions.

Where the last post described a “fan in” scenario, this is effectively the opposite use case: a “fan out” scenario.

(more…)

Using Mirror Maker 2 with IBM Event Streams to aggregate events across regions

Saturday, March 30th, 2024

This is the first in a series of blog posts sharing examples of ways to use Mirror Maker 2 with IBM Event Streams.

Mirror Maker 2 is a powerful and flexible tool for moving Kafka events between Kafka clusters, but sometimes I feel like this can be forgotten if we only talk about it in the context of disaster recovery.

In these posts, I want to inspire you to think about other ways you could use Mirror Maker 2. The best way to learn about what is possible is to play with it for yourself, so with these posts I’ll include a script to create a demonstration of the scenario.

For this first post, I’ll look at using Mirror Maker to aggregate events across clusters in multiple regions.

(more…)

Getting demo events onto IBM Event Streams topics in a hurry

Wednesday, March 13th, 2024

Sharing a couple of tips for quick-and-dirty demo setups.

I often need to put together demos of IBM Event Automation without much notice. The starting point is almost always needing to get a bunch of interesting events onto a Kafka topic.

What I need is a jumping-off point to illustrate the benefit of sharing streams of events in Event Endpoint Management, or the types of processing you can do in Event Processing. And to do that, I need a topic with events on them that will look interesting or relevant to who I’m demo’ing to.

If I’ve got time to do this properly, I’ll setup a generator that will give me a continuous stream of randonly-generated events (example). But if I’m in a hurry, I’ll use the REST Producer API and do something like this instead.

(more…)

Processing XML with Kafka Connect

Monday, December 11th, 2023

In this tutorial, I’ll share examples of how to process XML data at various points in a Kafka Connect pipeline, using a new plugin from IBM Event Streams.

You can assemble a Kafka Connect pipeline in a huge number of ways, so I’m not going to attempt an exhaustive list here. Instead, I’ve come up with eight examples that are illustrative of the sort of use cases you can satisfy.

I’ll summarise and link to my different examples here, so you can jump straight to the one that sounds the closest to your use case:

(more…)

What can you do with IBM Event Automation?

Sunday, July 9th, 2023

This is IBM Event Automation : a new product we released last month to help our clients create event driven solutions.

I’ve written a 200-word summary of what IBM Event Automation is, but in this post I wanted to dive a little bit deeper and show what it can do.

(more…)

What is IBM Event Automation?

Tuesday, July 4th, 2023

A summary of IBM Event Automation in under 200 words.

Last week was the first release of IBM Event Automation. I’ve been asked what it’s all about, so I thought it’d be helpful to have a brief summary to point people at.

IBM Event Automation is a collection of three components, designed to help companies be productive in creating event-driven solutions.

(more…)

Using client quotas with IBM Event Streams

Sunday, February 26th, 2023

In this post, I want to highlight a feature that I often see under-used in IBM Event Streams, and show how you can easily give it a try.

Kafka can enforce quotas to limit the impact that client applications can have on your cluster. To quote the Kafka documentation:

It is possible for producers and consumers to produce/consume very high volumes of data or generate requests at a very high rate and thus monopolize broker resources, cause network saturation and generally DOS other clients and the brokers themselves.

Having quotas protects against these issues and is all the more important in large multi-tenant clusters where a small set of badly behaved clients can degrade user experience for the well behaved ones.

In fact, when running Kafka as a service this even makes it possible to enforce API limits according to an agreed upon contract.

(more…)

Running IBM Event Streams on a laptop (sort of)

Friday, December 23rd, 2022

How to run a tiny local Kafka cluster using IBM Event Streams images

For local development on Kafka projects, I always run the public open source builds of ZooKeeper and Kafka as Java processes directly on my laptop (similar to steps described in the Apache Kafka Quickstart).

But for a project this week, I needed to verify something with the distribution of Kafka that comes with IBM Event Streams.

I used a simple Docker Compose setup for this. I’ll use this post to share how I did it.

(more…)