IBM Event Streams is the distributed streaming real-time data platform Apache Kafka, from IBM.
Node-RED is a visual flow-based development tool, with nodes that you drag and drop onto a canvas and wire together. It’s useful for loads of tasks, such as quick and flexible prototyping.
In this post, I’ll show how Event Streams and Node-RED work well together. You can use Node-RED to quickly and easily create flows that consume messages from Kafka topics, or that process events from different sources and produce the output to Kafka topics.
I’ll go through:
- Getting the Event Streams connection info you’ll need for Node-RED
- Adding the Kafka nodes to Node-RED
- Creating a flow that consumes messages from Kafka topics
- Creating a flow that produces messages to Kafka topics
Step 1 – Getting connection info
Here is the Event Streams UI. I’ve created a couple of topics, one for each of the flows I’ll set up.
I need to get connection info for Node-RED to connect to my Kafka cluster. The first step is to click on Connect to this cluster.
This brings up a connection panel with all the info I need. I need the bootstrap server address displayed there to tell Node-RED where to connect to, and to download the PEM version of the certificate so I can make the connection secure.
I also need to create an API key using the wizard on the right.
I could create two separate API keys – one with permission to consume messages for my first Node-RED flow, and a second with permission to produce messages for my second Node-RED flow. But this is just a quick silly demo and I’m lazy, so I created a single API key that can be used to do both.
Once I’ve got the API key, all the other info needed can be copied from the sample properties list.
Step 2 – Adding Kafka nodes to Node-RED
This is the Node-RED UI. Nodes are in the palette on the left. In addition to the nodes that are available by default out of the box, there is a rich ecosystem of nodes contributed by the Node-RED community.
There are lots of Kafka options available. For this post, I used node-red-contrib-rdkafka-secure but there are several other options available.
Installing this module adds a few new Kafka nodes to your Node-RED workspace.
The Event Streams connection info and certificate for my cluster from the previous step is all that is needed to configure them.
The bootstrap server address goes into the Broker List box, the location of the PEM certificate file goes in the SSL CA location box, and the rest of the values are copied from the connection info displayed in the Event Streams UI.
The Kafka nodes in the Node-RED palette are now able to connect to my Event Streams Kafka cluster.
Step 3 – Using Node-RED to consume Kafka messages
A flow to consume Kafka messages starts with a Kafka input node. You configure it with the name of the topic you want the flow to receive messages from.
For my quick demo flow, I’ve wired together a few simple nodes. The payload of the message will be run through Google Translate to translate into English. And finally, the translated message text will be sent in an SMS to my phone.
Yeah, I picked a couple of nodes pretty much at random. 🙂
To test, I just need to produce a message to the topic that the flow is consuming from. (I’m using kaf to do that).
It worked!
Okay, I was sort of hoping that sending “Bonjour le monde” would come through as “Hello world” but never mind – you get the idea.
Step 4 – Using Node-RED to produce Kafka messages
Another quick demo to show how you can use Node-RED to send Kafka messages, too.
I’ve added another flow to receive messages from Twitter (any messages that mention “IBM”), and send them to the Kafka topic “IBM.TWEETS”.
That’s all I need to see messages from Twitter start to appear on my Kafka topic.
You can also see the Kafka messages in the Event Streams UI. And clicking on one of the messages displays the tweet in the message browser.
Neither of these are particularly sensible Kafka demos. But the point was it was super quick and easy to wire together a few different services, making Node-RED a great way to visually prototype things that interact with Event Streams.
Tags: apachekafka, ibmeventstreams, kafka, nodered