{"id":4799,"date":"2022-12-23T16:25:59","date_gmt":"2022-12-23T16:25:59","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4799"},"modified":"2022-12-23T16:34:29","modified_gmt":"2022-12-23T16:34:29","slug":"running-ibm-event-streams-on-a-laptop-sort-of","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4799","title":{"rendered":"Running IBM Event Streams on a laptop (sort of)"},"content":{"rendered":"<p><strong>How to run a tiny local Kafka cluster using IBM Event Streams images<\/strong><\/p>\n<p>For local development on Kafka projects, I always run the public open source builds of ZooKeeper and Kafka as Java processes directly on my laptop (similar to steps described in the <a href=\"https:\/\/kafka.apache.org\/quickstart\">Apache Kafka Quickstart<\/a>).<\/p>\n<p>But for a project this week, I needed to verify something with the distribution of Kafka that comes with <a href=\"https:\/\/ibm.github.io\/event-streams\">IBM Event Streams<\/a>.<\/p>\n<p>I used a simple <a href=\"https:\/\/docs.docker.com\/compose\/\">Docker Compose<\/a> setup for this. I&#8217;ll use this post to share how I did it.<\/p>\n<p><!--more-->Here is the <code style=\"background-color: #ffffc0; font-weight: 600; color: #770000;\">docker-compose.yaml<\/code> file:<\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/c9f78a85872f6c778672cd68cbb69c94.js\"><\/script><br \/>\n<small><a href=\"https:\/\/gist.github.com\/dalelane\/c9f78a85872f6c778672cd68cbb69c94\">docker-compose.yaml on gist.github.com<\/a><\/small><\/p>\n<p>Note that you&#8217;ll need to <code style=\"background-color: #ffffc0; font-weight: 600; color: #770000;\">docker login<\/code> first before you can pull the container images:<\/p>\n<pre style=\"max-height: 250px; max-width: 90%; padding: 10px; overflow: scroll; overflow-x: scroll; border: thin black solid; font-size: smaller; background-color: #eeeeee; color: black; margin-left: 16px;\">docker login cp.icr.io --username cp --password YOUR_ENTITLEMENT_KEY<\/pre>\n<p>(You can get your entitlement key from <a href=\"https:\/\/myibm.ibm.com\/products-services\/containerlibrary\">IBM Container Library<\/a>.)<\/p>\n<p>Once you&#8217;ve got that, you just need to run <code style=\"background-color: #ffffc0; font-weight: 600; color: #770000;\">docker-compose up<\/code><\/p>\n<p>This gives you a tiny one-broker Kafka cluster running locally with ephemeral storage.<\/p>\n<p>I&#8217;ve included port-forwarding, so Kafka app running on your host machine can access the cluster at <code style=\"background-color: #ffffc0; font-weight: 600; color: #770000;\">localhost:9092<\/code>.<\/p>\n<p>For example:<\/p>\n<pre style=\"max-height: 250px; max-width: 90%; padding: 10px; overflow: scroll; overflow-x: scroll; border: thin black solid; font-size: smaller; background-color: #eeeeee; color: black; margin-left: 16px;\">kafka-topics.sh --bootstrap-server localhost:9092 --topic MY.TOPIC --create\n\nkafka-console-consumer.sh --bootstrap-server localhost:9092 --topic MY.TOPIC\n\nkafka-console-producer.sh --bootstrap-server localhost:9092 --topic MY.TOPIC<\/pre>\n<p>Or if you want to do <strong>everything<\/strong> from Docker containers, you can access it from other containers in the Docker Compose network at <code style=\"background-color: #ffffc0; font-weight: 600; color: #770000;\">kafka:29092<\/code>.<\/p>\n<p>For example:<\/p>\n<pre style=\"max-height: 250px; max-width: 90%; padding: 10px; overflow: scroll; overflow-x: scroll; border: thin black solid; font-size: smaller; background-color: #eeeeee; color: black; margin-left: 16px;\">docker-compose exec kafka \/bin\/bash -c \\\n    \".\/bin\/kafka-topics.sh --bootstrap-server kafka:29092 --topic MY.TOPIC --create\"\n\ndocker-compose exec kafka \/bin\/bash -c \\\n    \".\/bin\/kafka-console-consumer.sh --bootstrap-server kafka:29092 --topic MY.TOPIC\"\n\ndocker-compose exec kafka \/bin\/bash -c \\\n    \".\/bin\/kafka-console-producer.sh --bootstrap-server kafka:29092 --topic MY.TOPIC\"<\/pre>\n<p>A couple of caveats:<\/p>\n<p>1) Although this is using the same images and compiled\/built code as IBM Event Streams, running the images in this way skips a <strong>lot<\/strong> of the configuration and setup that happens in a real Event Streams deployment (such as setting up auth and encryption, enabling metrics, attaching additional agents, etc.). Running this Docker Compose setup does not get you a hardened, secured Kafka cluster. It&#8217;s a tiny single-broker unsecured cluster for development use.<\/p>\n<p>2) This does run on Apple Silicon, thanks to the wonders of emulation. (That&#8217;s why I added the <code>platform<\/code> attributes.) There is a delay on some operations, like creating new topics. But once I had things up and running, I didn&#8217;t notice any problems.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to run a tiny local Kafka cluster using IBM Event Streams images For local development on Kafka projects, I always run the public open source builds of ZooKeeper and Kafka as Java processes directly on my laptop (similar to steps described in the Apache Kafka Quickstart). But for a project this week, I needed [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,582,583,584],"class_list":["post-4799","post","type-post","status-publish","format-standard","hentry","category-code","tag-apachekafka","tag-eventstreams","tag-ibmeventstreams","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4799","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4799"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4799\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4799"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4799"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4799"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}