Posts Tagged ‘apachekafka’

Connecting App Connect Enterprise to Event Endpoint Management

Friday, October 20th, 2023

Configuring IBM App Connect Enterprise to consume messages from Kafka topics in IBM Event Endpoint Management requires careful configuration. In this post, I’ll share the steps I use that help me to avoid missing any required values.

If this sounds familiar, it might be because I wrote a post like this about using App Connect Enterprise to work with topics from Event Streams. People seem to have found that post useful, so I thought I’d do something similar for topics in Event Endpoint Management this time.

To illustrate this, I’ll create a simple App Connect flow that consumes messages from a Kafka topic and publish them to an MQTT topic.

The key to getting this to work correctly first time is to make sure that values are accurately copied from Event Endpoint Management to App Connect.

To help with this, I use a grid like the one below.

The instructions in this post start with Event Endpoint Management, and explain how to populate the grid with the information you need.

Then the instructions will switch to App Connect, and explain how to use the values in the grid to set up your App Connect flow.

What this is Values you will see in my screenshots Your value
A Topic name
DEMO.ACE
B Bootstrap address
my-eem-gateway-ibm-egw-rt-event-automation.apps.dalelane.cp.fyre.ibm.com:443
C SASL mechanism
PLAIN
D SASL config
org.apache.kafka.common.security.plain.PlainLoginModule required;
E Security protocol
SASL_SSL
F Certificate
eem-cert.jks
G Certificate password
STOREPASSW0RD
H Username
eem-9c8fc5d9-fddd-48dd-ab41-e062214166e5
I Password
dd08a1fc-99be-4931-8059-70aef88c1f0c
J Policy project name
demo-policies
K Policy name
demo-eem-policy
L Security identity name
eem-credentials
M Truststore identity name
eem-truststore

(more…)

Understanding windows in Event Processing

Wednesday, October 11th, 2023

Event Processing (one of the new capabilities of IBM Event Automation) makes it easy to perform stateful processing of streams of events from Kafka topics. In this post, I want to drill down a little into how windowed processing behaves.

I’ve enjoyed the chance to see the solutions that people have started to build with the tool. As part of this, I’ve been helping several people to understand the results produced by the event processing flows they’ve made.

These often started with different questions, such as:

  • Why hasn’t my flow produced any results?
  • Why isn’t my one-minute window producing one result every minute?
  • Why did the last one event on my Kafka topic cause results to be produced for several different windows?
  • etc.

However, these are often symptoms of a single common question: how windowed processing operations behave.

I’ve tried to come up with simple ways to demonstrate how it works, so in this post I want to share how I’m currently explaining it.


youtu.be/x_r6GNZmsd4

What can you do with IBM Event Automation?

Sunday, July 9th, 2023

This is IBM Event Automation : a new product we released last month to help our clients create event driven solutions.

I’ve written a 200-word summary of what IBM Event Automation is, but in this post I wanted to dive a little bit deeper and show what it can do.

(more…)

What is IBM Event Automation?

Tuesday, July 4th, 2023

A summary of IBM Event Automation in under 200 words.

Last week was the first release of IBM Event Automation. I’ve been asked what it’s all about, so I thought it’d be helpful to have a brief summary to point people at.

IBM Event Automation is a collection of three components, designed to help companies be productive in creating event-driven solutions.

(more…)

An introduction to Kafka Connect and Kafka Streams using Xbox

Sunday, June 18th, 2023

This is a talk I gave at Kafka Summit last month. It was an introduction to the Java APIs for Kafka Connect and Kafka Streams, using data from Xbox to bring the examples to life.


Confluent require personal details to watch recordings from Kafka Summit – sorry

(more…)

Using Xbox to get started with Kafka Connect & Kafka Streams

Wednesday, May 17th, 2023

It’s easy for developers who aren’t immersed in all-things-Kafka to assume that “Apache Kafka” just means an event backbone: something that hosts topics (and perhaps the client libraries to produce and consume messages using those topics). But Kafka is more than that. It is an ecosystem of tools that enables a complete event-streaming application.

That was the premise of this talk, recorded at Devoxx UK, which I gave to a room of Java developers. I introduced them to two other bits of Kafka: Kafka Connect (for getting data in and out of Kafka topics from external systems) and Kafka Streams (for developing stream processing applications).

Because they were Java developers, I thought the best way to give them a flavour of these tools was to show them the APIs, and walk through an example solution made using the APIs.

The example solution used Kafka tools to process data from Xbox – mostly because I’m a gamer and it made for a fun, if silly, demo.


recording of the talk at youtu.be/rvVjI1NXswU

(more…)

Using Apache Kafka with IBM MQ using Kafka Connect

Thursday, April 20th, 2023

A recording of a demo walkthrough I did about using the Kafka Connect MQ connectors to flow messages between IBM MQ and Apache Kafka.

A few weeks ago, I presented a session at TechCon about IBM MQ and Apache Kafka with David Ware. I spent most of my time running through how to use Kafka Connect with IBM MQ, with a few demos showing different ways to setup and run the kafka-connect-mq-source Connector.

My demos start at around 20 minutes in, but you should listen to David give the context first!

Using client quotas with IBM Event Streams

Sunday, February 26th, 2023

In this post, I want to highlight a feature that I often see under-used in IBM Event Streams, and show how you can easily give it a try.

Kafka can enforce quotas to limit the impact that client applications can have on your cluster. To quote the Kafka documentation:

It is possible for producers and consumers to produce/consume very high volumes of data or generate requests at a very high rate and thus monopolize broker resources, cause network saturation and generally DOS other clients and the brokers themselves.

Having quotas protects against these issues and is all the more important in large multi-tenant clusters where a small set of badly behaved clients can degrade user experience for the well behaved ones.

In fact, when running Kafka as a service this even makes it possible to enforce API limits according to an agreed upon contract.

(more…)