{"id":3741,"date":"2019-07-06T20:59:35","date_gmt":"2019-07-06T20:59:35","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=3741"},"modified":"2019-07-07T10:50:35","modified_gmt":"2019-07-07T10:50:35","slug":"getting-started-with-openwhisk-and-kafka","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=3741","title":{"rendered":"Getting started with OpenWhisk and Kafka"},"content":{"rendered":"<p><a href=\"https:\/\/openwhisk.apache.org\/\">Apache OpenWhisk<\/a> (and serverless platforms in general) are a great way to host and manage code that you want to run in response to events.<br \/>\n<a href=\"https:\/\/kafka.apache.org\/\">Apache Kafka<\/a> topics are a great source of events. <\/p>\n<p><strong>In this post, I&#8217;ll run through a super simple beginner&#8217;s guide to writing code for OpenWhisk that processes events on your Kafka topics.<\/strong><\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-78.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-78.png\"\/><\/a><\/p>\n<p><!--more-->To start with, I needed to get OpenWhisk running on my laptop. There are a variety of deployment options. You can install it onto bare metal using the Ansible installer. If you have Kubernetes or Mesos or Docker Compose environments, there are installers for each of these, too.<\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-73.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-73.png\"\/><\/a><\/p>\n<p>For this post, I used the Docker Compose option.<\/p>\n<p>An easy to run OpenWhisk on your laptop is to use the installer available in the <a href=\"https:\/\/github.com\/apache\/incubator-openwhisk-devtools\">devtools repo<\/a>. <\/p>\n<p>That gives you a small local development environment that you can comfortably run on your own machine. <\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/DQ8MxX04bV8\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">git clone https:\/\/github.com\/apache\/incubator-openwhisk-devtools.git\r\ncd incubator-openwhisk-devtools\/docker-compose\r\nmake quick-start<\/pre>\n<p>You can clone the OpenWhisk devtools repo, cd into the docker-compose folder and run make quick-start. <\/p>\n<p>The video above shows you what to expect. It takes a few minutes. It&#8217;s a start-then-go-get-a-cup-of-coffee sort of job. <\/p>\n<p>Most of that time is pulling the Docker images for the various components that make up OpenWhisk. This means it&#8217;ll be a lot quicker the next time you have to do it!<\/p>\n<p>Once it&#8217;s pulled them, it configures them and starts them up. <\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-75.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-75.png\"\/><\/a><\/p>\n<p>Events are emitted by event providers. OpenWhisk comes with a few options for events from database triggers, HTTP calls, Kafka topics, scheduled events. <\/p>\n<p>For this post, I wanted my code invoked in response to messages on Kafka topics, so that means I used the Kafka event provider. <\/p>\n<p>The local devtools development install I used includes an installer for a few different providers, including Kafka. <\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/jZYZmA9HMhc\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">cd incubator-openwhisk-devtools\/docker-compose\r\nmake create-provider-cloudant\r\nmake create-provider-kafka<\/pre>\n<p>The video snippet above shows how you can install the pre-built Kafka event provider. <\/p>\n<p>If you want to build it yourself (or you want to add the Kafka event provider to an OpenWhisk that isn&#8217;t the local laptop dev environment deployment), you can get it by cloning the <a href=\"https:\/\/github.com\/apache\/incubator-openwhisk-package-kafka\">openwhisk-package-kafka<\/a> repo, and running the <code>installKafka<\/code> command, pointing it at your OpenWhisk instance. <\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/WhIUL-Hv5fw\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">git clone https:\/\/github.com\/apache\/incubator-openwhisk-package-kafka.git\r\ncd incubator-openwhisk-package-kafka\r\nexport OPENWHISK_AUTH=$OPENWHISK_HOME\/ansible\/files\/auth.whisk.system\r\n.\/installKafka.sh $OPENWHISK_AUTH $SYSTEM_IP_ADDR \\\r\n    http:\/\/$SYSTEM_IP_ADDR:5984 \\    \r\n    local_ \\\r\n    $SYSTEM_IP_ADDR<\/pre>\n<p>The video snippet above shows how I added the <code>kafkaFeed<\/code> event provider as an option for triggers that I create in my OpenWhisk instance.<\/p>\n<p>Now that I\u2019ve added the Kafka event provider, it&#8217;s time to show how to use it.<\/p>\n<p>In this post, I&#8217;ll create this demo.<\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-78.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-78.png\"\/><\/a><\/p>\n<p>I&#8217;ll show how to use the Kafka event provider to define a trigger in OpenWhisk based on a Kafka topic, and use that trigger to invoke a function.<\/p>\n<p>My function takes the message from the Kafka topic and sends that to Slack. <\/p>\n<p>It&#8217;s not something you\u2019d probably want to do, but it makes for a very quick and simple demo to give you the idea. <\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-79.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-79.png\"\/><\/a><\/p>\n<p>The first step is to work out where my events are going to come from. <\/p>\n<p>The events for this post are messages produced to a Kafka topic called <code>MY.EVENTS<\/code>.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/oQX37Uz5CKY\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">kafka-topics.sh --create --topic MY.EVENTS \\\r\n    --replication-factor 1 --partitions 1 \\\r\n    --zookeeper $SYSTEM_IP_ADDR:3181<\/pre>\n<p>The video snippet above shows how I created a simple single partition Kafka topic. <\/p>\n<p>(The Kafka broker I&#8217;m using for this post is running on my laptop as well. As there is a Kafka cluster that runs as part of OpenWhisk, I&#8217;m using non-default port numbers for my own Kafka cluster, so the port numbers don&#8217;t clash. That&#8217;s why my ZooKeeper is on an unusual port number. If you&#8217;re using a Kafka cluster that isn&#8217;t on the same system as your OpenWhisk, you won&#8217;t need to worry about that.)<\/p>\n<p>This gave me a Kafka topic to use as a source of events.<\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-81.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-81.png\"\/><\/a><\/p>\n<p>Next I needed to define a trigger. <\/p>\n<p>For this post, I called it <code>myKafkaTrigger<\/code> to fire any time a message arrives on the <code>MY.EVENTS<\/code> topic.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/nYGlyOyxa10\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">wsk trigger create myKafkaTrigger \\\r\n    -f \/whisk.system\/messaging\/kafkaFeed \\\r\n    -p brokers '[\"9.174.23.208:3092\"]' \\\r\n    -p topic MY.EVENTS<\/pre>\n<p>The video snippet above shows how I created a trigger using the <code>wsk<\/code> command line tool. <\/p>\n<p>The syntax is:<br \/>\n<code>wsk trigger create<\/code><br \/>\nfollowed by the name of my trigger<br \/>\nthen <code>-f<\/code> and the name of the event provider to use (I&#8217;m using the <code>kafkaFeed<\/code> event provider I installed before)<br \/>\nthen <code>\u2013p<\/code> for each of the parameters I need to specify.<\/p>\n<p>For this post, I just gave it the bootstrap address and topic name, but if you have a Kafka cluster with authentication enabled, you&#8217;ll need to add extra parameters for that as well. <\/p>\n<p><em>Notice that even though the Kafka broker I want to use as a trigger is on the same machine as where I&#8217;m running OpenWhisk, I&#8217;m using the external IP address for my laptop, not `localhost`. This is because the OpenWhisk components will be running in Docker containers, so when it runs in there, localhost would refer to the container not the host, and it wouldn&#8217;t work.<\/em> <\/p>\n<p>Now I had my topic and defined a trigger that will fire. <\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-84.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-84.png\"\/><\/a><\/p>\n<p>The next thing to do is to implement a function. <\/p>\n<p>For this post, I implemented my function in JavaScript to be executed by Node.js. <\/p>\n<p>And I called it <code>sendToSlack<\/code>.<\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-85.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-85.png\"\/><\/a><\/p>\n<p>This is the implementation. <\/p>\n<p>It gets the messages from the Kafka trigger, get the text out of them, and then posts them to Slack using Slack&#8217;s incoming webhook REST API. <\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/5YCo_YZSBBM\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">wsk action create sendToSlack slack-action.js<\/pre>\n<p>The video snippet above shows how I created the action using my JavaScript implementation. <\/p>\n<p>The syntax is:<br \/>\n<code>wsk action create<\/code><br \/>\nfollowed by the name of the action I want to create <code>sendToSlack<\/code><br \/>\nthen the name of the file that contains the implementation <code>slack-action.js<\/code><\/p>\n<p>Now I had my trigger defined and I created the function with the action that will run. <\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-88.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-88.png\"\/><\/a><\/p>\n<p>The next step is to create the rule that will bind these together.<\/p>\n<p>I called it <code>slackOnKafkaMessages<\/code>.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/uOIsPYQl2aA\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">wsk rule create slackOnKafkaMessages \\ \r\n    myKafkaTrigger \\\r\n    sendToSlack<\/pre>\n<p>The video snippet above shows how I created a rule using the <code>wsk<\/code> command line. <\/p>\n<p>The syntax is:<br \/>\n<code>wsk rule create<\/code><br \/>\nfollowed by the name of my rule<br \/>\nthen the name of the trigger<br \/>\nand finally the name of the action that I want invoked<\/p>\n<p>That&#8217;s all I needed to do. The demo is ready to go. <\/p>\n<p><a href=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-88.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" style=\"border: thin black solid\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/190706-openwhisk-kafka\/190706-openwhisk-88.png\"\/><\/a><\/p>\n<p>To recap, I send a message to the Kafka topic <code>MY.EVENTS<\/code>.<br \/>\nMy trigger, called <code>myKafkaTrigger<\/code>, fires in response to the message.<br \/>\nThe rule <code>slackOnKafkaMessages<\/code> causes my action to be invoked.<br \/>\nMy action is a JavaScript function that makes a REST call to Slack. <\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/KWnar-YBNjs\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p>The video snippet above shows all of this in action.<\/p>\n<p>I used the Kafka console producer to send messages to the Kafka topic, and they showed up pretty much instantly on my Slack channel. <\/p>\n<p>As I said, this isn&#8217;t the most useful demo application in the world, but it shows how simple it is to deploy a function to OpenWhisk and configure it to be run in response to Kafka messages.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Apache OpenWhisk (and serverless platforms in general) are a great way to host and manage code that you want to run in response to events. Apache Kafka topics are a great source of events. In this post, I&#8217;ll run through a super simple beginner&#8217;s guide to writing code for OpenWhisk that processes events on your [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":3765,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,584,592],"class_list":["post-3741","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-apachekafka","tag-kafka","tag-openwhisk"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3741","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3741"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3741\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/3765"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3741"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3741"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3741"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}