How to run a tiny local Kafka cluster using IBM Event Streams images
For local development on Kafka projects, I always run the public open source builds of ZooKeeper and Kafka as Java processes directly on my laptop (similar to steps described in the Apache Kafka Quickstart).
But for a project this week, I needed to verify something with the distribution of Kafka that comes with IBM Event Streams.
I used a simple Docker Compose setup for this. I’ll use this post to share how I did it.
Here is the docker-compose.yaml file:
docker-compose.yaml on gist.github.com
Note that you’ll need to docker login first before you can pull the container images:
docker login cp.icr.io --username cp --password YOUR_ENTITLEMENT_KEY
(You can get your entitlement key from IBM Container Library.)
Once you’ve got that, you just need to run docker-compose up
This gives you a tiny one-broker Kafka cluster running locally with ephemeral storage.
I’ve included port-forwarding, so Kafka app running on your host machine can access the cluster at localhost:9092.
For example:
kafka-topics.sh --bootstrap-server localhost:9092 --topic MY.TOPIC --create kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic MY.TOPIC kafka-console-producer.sh --bootstrap-server localhost:9092 --topic MY.TOPIC
Or if you want to do everything from Docker containers, you can access it from other containers in the Docker Compose network at kafka:29092.
For example:
docker-compose exec kafka /bin/bash -c \
"./bin/kafka-topics.sh --bootstrap-server kafka:29092 --topic MY.TOPIC --create"
docker-compose exec kafka /bin/bash -c \
"./bin/kafka-console-consumer.sh --bootstrap-server kafka:29092 --topic MY.TOPIC"
docker-compose exec kafka /bin/bash -c \
"./bin/kafka-console-producer.sh --bootstrap-server kafka:29092 --topic MY.TOPIC"
A couple of caveats:
1) Although this is using the same images and compiled/built code as IBM Event Streams, running the images in this way skips a lot of the configuration and setup that happens in a real Event Streams deployment (such as setting up auth and encryption, enabling metrics, attaching additional agents, etc.). Running this Docker Compose setup does not get you a hardened, secured Kafka cluster. It’s a tiny single-broker unsecured cluster for development use.
2) This does run on Apple Silicon, thanks to the wonders of emulation. (That’s why I added the platform attributes.) There is a delay on some operations, like creating new topics. But once I had things up and running, I didn’t notice any problems.
Tags: apachekafka, eventstreams, ibmeventstreams, kafka