dale lane

In this post, I want to share an example of handling bespoke structured messages with the Kafka Connect MQ Source Connector.

The MQ Source Connector gets data from MQ messages and produces it as events on Kafka topics. The default record builder makes a copy of the data as-is. For example, this can mean taking a JMS TextMessage from MQ and producing a string to Kafka. Or it can mean taking a JMS BytesMessage from MQ and producing a byte array to Kafka.

In my last post, I showed an example of using the XML record builder, to read XML documents from MQ and turn them into structured Kafka Connect records. From this point, I could choose the format I want the data to be produced to Kafka in (e.g. JSON or Avro) by choosing an appropriate value converter (e.g. org.apache.kafka.connect.json.JsonConverter or io.apicurio.registry.utils.converter.AvroConverter).

But what if your MQ messages have a custom structure, but you still want Kafka Connect to be able to parse your messages and output them to Kafka in any format of your choice?

In that case, you need to use a record builder that can correctly parse your MQ messages. In this post, I’ll explain what that means, show you how to create one, and share a sample you can use to get started.

(more…)

In this post, I’ll walk through a demo of using IBM Event Processing to create an Apache Flink job that calculates summaries of messages from IBM MQ queues.

This is a high-level overview of the demo:

• A JMS/Jakarta application puts XML messages onto an MQ queue
• A JSON version of these messages is copied onto a Kafka topic
• The messages are processed by a Flink job, which outputs JSON results onto a Kafka topic
• An XML version of the results are copied onto an MQ queue
• The results are received by a JMS/Jakarta application

I’ve added instructions for how you can create a demo like this for yourself to my demos repo on Github.

The rest of this post is a walkthrough and explanation of how it all works.

(more…)

aka “Who wants a Mario alarm clock?”

In this post, I want to share a quick demo of using Event Processing to process social media posts.

Background

A fun surprise from Nintendo today: they’ve introduced a new product! “Alarmo” is a game-themed alarm clock, with some interesting gesture recognition features.

I was (unsurprisingly!) tempted…

But that got me wondering how the rest of the Internet was reacting.

In this post, I want to share a (super-simple!) demo for how to look at this – using IBM Event Processing to create an Apache Flink job that looks at the sentiment of social media posts about this unusual new product.

(more…)