Apache OpenWhisk (and serverless platforms in general) are a great way to host and manage code that you want to run in response to events.
Apache Kafka topics are a great source of events.
In this post, I’ll run through a super simple beginner’s guide to writing code for OpenWhisk that processes events on your Kafka topics.
To start with, I needed to get OpenWhisk running on my laptop. There are a variety of deployment options. You can install it onto bare metal using the Ansible installer. If you have Kubernetes or Mesos or Docker Compose environments, there are installers for each of these, too.
For this post, I used the Docker Compose option.
An easy to run OpenWhisk on your laptop is to use the installer available in the devtools repo.
That gives you a small local development environment that you can comfortably run on your own machine.
git clone https://github.com/apache/incubator-openwhisk-devtools.git cd incubator-openwhisk-devtools/docker-compose make quick-start
You can clone the OpenWhisk devtools repo, cd into the docker-compose folder and run make quick-start.
The video above shows you what to expect. It takes a few minutes. It’s a start-then-go-get-a-cup-of-coffee sort of job.
Most of that time is pulling the Docker images for the various components that make up OpenWhisk. This means it’ll be a lot quicker the next time you have to do it!
Once it’s pulled them, it configures them and starts them up.
Events are emitted by event providers. OpenWhisk comes with a few options for events from database triggers, HTTP calls, Kafka topics, scheduled events.
For this post, I wanted my code invoked in response to messages on Kafka topics, so that means I used the Kafka event provider.
The local devtools development install I used includes an installer for a few different providers, including Kafka.
cd incubator-openwhisk-devtools/docker-compose make create-provider-cloudant make create-provider-kafka
The video snippet above shows how you can install the pre-built Kafka event provider.
If you want to build it yourself (or you want to add the Kafka event provider to an OpenWhisk that isn’t the local laptop dev environment deployment), you can get it by cloning the openwhisk-package-kafka repo, and running the installKafka command, pointing it at your OpenWhisk instance.
git clone https://github.com/apache/incubator-openwhisk-package-kafka.git
cd incubator-openwhisk-package-kafka
export OPENWHISK_AUTH=$OPENWHISK_HOME/ansible/files/auth.whisk.system
./installKafka.sh $OPENWHISK_AUTH $SYSTEM_IP_ADDR \
http://$SYSTEM_IP_ADDR:5984 \
local_ \
$SYSTEM_IP_ADDR
The video snippet above shows how I added the kafkaFeed event provider as an option for triggers that I create in my OpenWhisk instance.
Now that I’ve added the Kafka event provider, it’s time to show how to use it.
In this post, I’ll create this demo.
I’ll show how to use the Kafka event provider to define a trigger in OpenWhisk based on a Kafka topic, and use that trigger to invoke a function.
My function takes the message from the Kafka topic and sends that to Slack.
It’s not something you’d probably want to do, but it makes for a very quick and simple demo to give you the idea.
The first step is to work out where my events are going to come from.
The events for this post are messages produced to a Kafka topic called MY.EVENTS.
kafka-topics.sh --create --topic MY.EVENTS \
--replication-factor 1 --partitions 1 \
--zookeeper $SYSTEM_IP_ADDR:3181
The video snippet above shows how I created a simple single partition Kafka topic.
(The Kafka broker I’m using for this post is running on my laptop as well. As there is a Kafka cluster that runs as part of OpenWhisk, I’m using non-default port numbers for my own Kafka cluster, so the port numbers don’t clash. That’s why my ZooKeeper is on an unusual port number. If you’re using a Kafka cluster that isn’t on the same system as your OpenWhisk, you won’t need to worry about that.)
This gave me a Kafka topic to use as a source of events.
Next I needed to define a trigger.
For this post, I called it myKafkaTrigger to fire any time a message arrives on the MY.EVENTS topic.
wsk trigger create myKafkaTrigger \
-f /whisk.system/messaging/kafkaFeed \
-p brokers '["9.174.23.208:3092"]' \
-p topic MY.EVENTS
The video snippet above shows how I created a trigger using the wsk command line tool.
The syntax is:
wsk trigger create
followed by the name of my trigger
then -f and the name of the event provider to use (I’m using the kafkaFeed event provider I installed before)
then –p for each of the parameters I need to specify.
For this post, I just gave it the bootstrap address and topic name, but if you have a Kafka cluster with authentication enabled, you’ll need to add extra parameters for that as well.
Notice that even though the Kafka broker I want to use as a trigger is on the same machine as where I’m running OpenWhisk, I’m using the external IP address for my laptop, not `localhost`. This is because the OpenWhisk components will be running in Docker containers, so when it runs in there, localhost would refer to the container not the host, and it wouldn’t work.
Now I had my topic and defined a trigger that will fire.
The next thing to do is to implement a function.
For this post, I implemented my function in JavaScript to be executed by Node.js.
And I called it sendToSlack.
This is the implementation.
It gets the messages from the Kafka trigger, get the text out of them, and then posts them to Slack using Slack’s incoming webhook REST API.
wsk action create sendToSlack slack-action.js
The video snippet above shows how I created the action using my JavaScript implementation.
The syntax is:
wsk action create
followed by the name of the action I want to create sendToSlack
then the name of the file that contains the implementation slack-action.js
Now I had my trigger defined and I created the function with the action that will run.
The next step is to create the rule that will bind these together.
I called it slackOnKafkaMessages.
wsk rule create slackOnKafkaMessages \
myKafkaTrigger \
sendToSlack
The video snippet above shows how I created a rule using the wsk command line.
The syntax is:
wsk rule create
followed by the name of my rule
then the name of the trigger
and finally the name of the action that I want invoked
That’s all I needed to do. The demo is ready to go.
To recap, I send a message to the Kafka topic MY.EVENTS.
My trigger, called myKafkaTrigger, fires in response to the message.
The rule slackOnKafkaMessages causes my action to be invoked.
My action is a JavaScript function that makes a REST call to Slack.
The video snippet above shows all of this in action.
I used the Kafka console producer to send messages to the Kafka topic, and they showed up pretty much instantly on my Slack channel.
As I said, this isn’t the most useful demo application in the world, but it shows how simple it is to deploy a function to OpenWhisk and configure it to be run in response to Kafka messages.
Tags: apachekafka, kafka, openwhisk