Using Node-RED with IBM Event Streams


Click to enlarge

IBM Event Streams is the distributed streaming real-time data platform Apache Kafka, from IBM.

Node-RED is a visual flow-based development tool, with nodes that you drag and drop onto a canvas and wire together. It’s useful for loads of tasks, such as quick and flexible prototyping.

In this post, I’ll show how Event Streams and Node-RED work well together. You can use Node-RED to quickly and easily create flows that consume messages from Kafka topics, or that process events from different sources and produce the output to Kafka topics.

I’ll go through:

  1. Getting the Event Streams connection info you’ll need for Node-RED
  2. Adding the Kafka nodes to Node-RED
  3. Creating a flow that consumes messages from Kafka topics
  4. Creating a flow that produces messages to Kafka topics

Step 1 – Getting connection info


Click to enlarge

Here is the Event Streams UI. I’ve created a couple of topics, one for each of the flows I’ll set up.

I need to get connection info for Node-RED to connect to my Kafka cluster. The first step is to click on Connect to this cluster.


Click to enlarge

This brings up a connection panel with all the info I need. I need the bootstrap server address displayed there to tell Node-RED where to connect to, and to download the PEM version of the certificate so I can make the connection secure.

I also need to create an API key using the wizard on the right.


Click to enlarge

I could create two separate API keys – one with permission to consume messages for my first Node-RED flow, and a second with permission to produce messages for my second Node-RED flow. But this is just a quick silly demo and I’m lazy, so I created a single API key that can be used to do both.


Click to enlarge

Once I’ve got the API key, all the other info needed can be copied from the sample properties list.

Step 2 – Adding Kafka nodes to Node-RED


Click to enlarge

This is the Node-RED UI. Nodes are in the palette on the left. In addition to the nodes that are available by default out of the box, there is a rich ecosystem of nodes contributed by the Node-RED community.


Click to enlarge

There are lots of Kafka options available. For this post, I used node-red-contrib-rdkafka-secure but there are several other options available.


Click to enlarge

Installing this module adds a few new Kafka nodes to your Node-RED workspace.

The Event Streams connection info and certificate for my cluster from the previous step is all that is needed to configure them.


Click to enlarge

The bootstrap server address goes into the Broker List box, the location of the PEM certificate file goes in the SSL CA location box, and the rest of the values are copied from the connection info displayed in the Event Streams UI.

The Kafka nodes in the Node-RED palette are now able to connect to my Event Streams Kafka cluster.

Step 3 – Using Node-RED to consume Kafka messages


Click to enlarge

A flow to consume Kafka messages starts with a Kafka input node. You configure it with the name of the topic you want the flow to receive messages from.


Click to enlarge

For my quick demo flow, I’ve wired together a few simple nodes. The payload of the message will be run through Google Translate to translate into English. And finally, the translated message text will be sent in an SMS to my phone.

Yeah, I picked a couple of nodes pretty much at random. 🙂


Click to enlarge

To test, I just need to produce a message to the topic that the flow is consuming from. (I’m using kaf to do that).


Click to enlarge

It worked!

Okay, I was sort of hoping that sending “Bonjour le monde” would come through as “Hello world” but never mind – you get the idea.

Step 4 – Using Node-RED to produce Kafka messages

Another quick demo to show how you can use Node-RED to send Kafka messages, too.


Click to enlarge

I’ve added another flow to receive messages from Twitter (any messages that mention “IBM”), and send them to the Kafka topic “IBM.TWEETS”.


Click to enlarge

That’s all I need to see messages from Twitter start to appear on my Kafka topic.


Click to enlarge

You can also see the Kafka messages in the Event Streams UI. And clicking on one of the messages displays the tweet in the message browser.


Click to enlarge

Neither of these are particularly sensible Kafka demos. But the point was it was super quick and easy to wire together a few different services, making Node-RED a great way to visually prototype things that interact with Event Streams.

Tags: , , ,

Leave a Reply