{"id":3664,"date":"2019-06-09T17:33:01","date_gmt":"2019-06-09T17:33:01","guid":{"rendered":"http:\/\/dalelane.co.uk\/blog\/?p=3664"},"modified":"2019-07-07T11:39:08","modified_gmt":"2019-07-07T11:39:08","slug":"using-kafkacat-and-kaf-with-ibm-event-streams","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=3664","title":{"rendered":"Using kafkacat and kaf with IBM Event Streams"},"content":{"rendered":"

IBM Event Streams<\/a> is IBM’s Kafka<\/a> offering. Naturally it comes with it’s own UI and CLI tools, but one of the great things about Apache Kafka is that it\u2019s not just a single thing from a single company – rather it is an active and diverse ecosystem, which means you\u2019ve got a variety of tools to choose from. <\/p>\n

I thought I’d try a couple of open source CLI tools, and share how to connect them and what they can do. <\/p>\n

First up, kafkacat<\/code>. <\/p>\n

<\/p>\n

kafkacat<\/h3>\n

I’ve used this before, but not for a while. <\/p>\n

To install<\/strong> it, I just used: <\/p>\n

brew install kafkacat<\/pre>\n
There are lots of other install options at github.com\/edenhill\/kafkacat<\/a>. <\/p>\n
To configure<\/strong> it, I created a file at ~\/.config\/kafkacat.conf<\/code> with the contents:<\/p>\n
bootstrap.servers=9.20.196.31:30885\r\nssl.ca.location=\/Users\/dalelane\/myfolder\/es-cert.pem\r\nsecurity.protocol=sasl_ssl\r\nsasl.mechanisms=PLAIN\r\nsasl.username=token\r\nsasl.password=xxxxx-MY-API-KEY-HERE-xxxxx<\/pre>\n
(I’ll explain where to get these config values for your cluster at the end of this post.)<\/em><\/p>\n
Once you’ve done that you can use kafkacat<\/code> to produce and consume messages<\/strong> in a variety of ways. <\/p>\n
You can produce messages from stdin using kafkacat -P<\/code>:<\/p>\n
kafkacat -P -t YOUR.TOPIC.NAME<\/pre>\n
Or produce a message with the contents of a file using:<\/p>\n
kafkacat -P -t YOUR.TOPIC.NAME your-file-name.txt<\/pre>\n
And include headers using:<\/p>\n
kafkacat -P -t YOUR.TOPIC.NAME -H \"headerkey=headervalue\" your-file-name.txt<\/pre>\n
Similarly, you can consume messages using kafkacat -C<\/code>:<\/p>\n
kafkacat -C -G your-consumer-group-id YOUR.TOPIC.NAME<\/pre>\n
And if you want to see the headers, you can customize the format like:<\/p>\n
kafkacat -C -G your-consumer-group-id -f 'headers:\\n%h\\nmessage:\\n%s' YOUR.TOPIC.NAME<\/pre>\n
There are tons of other options, but they’re well explained at github.com\/edenhill\/kafkacat<\/a> so I won’t reproduce it here. Suffice to say, you can control exactly how you produce or consume messages. <\/p>\n
But if all you want is a quick and easy way to send and receive messages, it’s good for that, too. <\/p>\n
And you can query for metadata about topics, with kafkacat -L<\/code>.<\/p>\n
$ kafkacat -L -t MY.TOPIC\r\nMetadata for MY.TOPIC (from broker -1: sasl_ssl:\/\/9.20.196.31:30885\/bootstrap):\r\n 3 brokers:\r\n  broker 0 at 9.20.196.31:31179 (controller)\r\n  broker 2 at 9.20.196.31:30011\r\n  broker 1 at 9.20.196.31:31053\r\n 1 topics:\r\n  topic \"MY.TOPIC\" with 3 partitions:\r\n    partition 0, leader 1, replicas: 1,0, isrs: 1,0\r\n    partition 1, leader 0, replicas: 0,2, isrs: 0,2\r\n    partition 2, leader 2, replicas: 2,1, isrs: 2,1<\/pre>\n
Next up, kaf<\/code>. <\/p>\n
<\/p>\n
kaf<\/h3>\n
This was a new one on me, that I first heard about at Kafka Summit last month<\/a>. <\/p>\n
To install<\/strong> it, I just used: <\/p>\n
curl https:\/\/raw.githubusercontent.com\/infinimesh\/kaf\/master\/godownloader.sh | BINDIR=$HOME\/bin bash<\/pre>\n
To configure<\/strong> it, I created a file at ~\/.kaf\/config<\/code> with the contents:<\/p>\n
current-cluster: mycluster\r\nclusters:\r\n- name: mycluster\r\n  brokers:\r\n  - 9.20.196.31:30885\r\n  SASL:\r\n    mechanism: PLAIN\r\n    username: token\r\n    password: xxxxx-MY-API-KEY-HERE-xxxxx\r\n  TLS:\r\n    cafile: \/Users\/dalelane\/myfolder\/es-cert.pem\r\n  security-protocol: SASL_SSL<\/pre>\n
(I’ll explain where to get these config values for your cluster at the end of this post.)<\/em><\/p>\n
Once you’ve done that you can use kaf<\/code> to manage your cluster, as well as produce and consume messages<\/strong>. <\/p>\n
You can get a list of topics:<\/p>\n
$ kaf topics\r\nNAME                 PARTITIONS   REPLICAS\r\n__consumer_offsets   50           3\r\ndale                 1            1\r\nTEST.TOPIC           1            3<\/pre>\n
You can query the properties of a topic:<\/p>\n
$ kaf topic describe dale\r\nName:        dale\r\nInternal:    false\r\nCompacted:   false\r\nPartitions:\r\n  Partition  High Watermark  Leader  Replicas  ISR\r\n  ---------  --------------  ------  --------  ---\r\n  0          16              2       [2]       [2]\r\nConfig:\r\n  Name                    Value    ReadOnly  Sensitive\r\n  ----                    -----    --------  ---------\r\n  cleanup.policy          delete   false     false\r\n  message.format.version  2.2-IV1  false     false<\/pre>\n
You can create a topic with:<\/p>\n
$ kaf topic create MY.TOPIC -p 3 -r 2\r\nCreated topic MY.TOPIC.<\/pre>\n
And a variety of similar admin tasks for topics and consumer groups. <\/p>\n
Like kafkacat<\/code>, you can also use kaf<\/code> to easily produce and consume messages from the command line. <\/p>\n
You can produce messages from stdin using:<\/p>\n
$ echo -n \"My message\" | kaf produce MY.TOPIC\r\nSent record to partition 0 at offset 0.<\/pre>\n
Or produce a message with the contents of a file using:<\/p>\n
$ cat your-file-name.txt | kaf produce MY.TOPIC\r\nSent record to partition 0 at offset 1.<\/pre>\n
Similarly, you can consume messages using:<\/p>\n
kaf consume MY.TOPIC<\/pre>\n
There aren’t as many options as kafkacat<\/code> has, but I found it a little more intuitive. And it’s got all the options you need to quickly and easily send and receive messages. You can read more about it at github.com\/birdayz\/kaf<\/a>. <\/p>\n
I regularly work with a variety of different clusters, and I really like the way that you can put the config for multiple clusters in the config file, give each one a friendly name, and then switch between them with kaf config use-cluster mycluster<\/code>. I think that’ll come in handy.<\/p>\n
Getting the config values<\/h3>\n
Finally, a quick few pointers for where I got the config values that I used above.<\/p>\n
Starting from the Event Streams UI…<\/p>\n
<\/a>
\nClick on the image for high-res version<\/small><\/p>\n
I clicked on Connect to this cluster<\/strong> on the right to bring up this side panel…<\/p>\n
<\/a>
\nClick on the image for high-res version<\/small><\/p>\n
Scrolling down shows the button I used to download the PEM certificate file.<\/p>\n
<\/a>
\nClick on the image for high-res version<\/small><\/p>\n
To the right is a small wizard for creating API keys… <\/p>\n
<\/a>
\nClick on the image for high-res version<\/small><\/p>\n
It walks you through a bunch of options, so you can decide exactly what actions the API key is allowed to take, and which topics it can use (or you can just keep picking “all”)<\/p>\n
Finally, the middle tab was where I got a helpful reminder for the rest of the config options to use.<\/p>\n
<\/a>
\nClick on the image for high-res version<\/small><\/p>\n
That was a quick play with kafkacat<\/code> and kaf<\/code>. What other tools should I be trying next?<\/p>\n","protected":false},"excerpt":{"rendered":"
IBM Event Streams is IBM’s Kafka offering. Naturally it comes with it’s own UI and CLI tools, but one of the great things about Apache Kafka is that it\u2019s not just a single thing from a single company – rather it is an active and diverse ecosystem, which means you\u2019ve got a variety of tools […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,4],"tags":[593,583,584],"class_list":["post-3664","post","type-post","status-publish","format-standard","hentry","category-code","category-ibm","tag-apachekafka","tag-ibmeventstreams","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3664","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3664"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3664\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3664"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3664"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3664"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}