Running IBM Event Streams on a laptop (sort of)

How to run a tiny local Kafka cluster using IBM Event Streams images

For local development on Kafka projects, I always run the public open source builds of ZooKeeper and Kafka as Java processes directly on my laptop (similar to steps described in the Apache Kafka Quickstart).

But for a project this week, I needed to verify something with the distribution of Kafka that comes with IBM Event Streams.

I used a simple Docker Compose setup for this. I’ll use this post to share how I did it.

Here is the docker-compose.yaml file:

docker-compose.yaml on

Note that you’ll need to docker login first before you can pull the container images:

docker login --username cp --password YOUR_ENTITLEMENT_KEY

(You can get your entitlement key from IBM Container Library.)

Once you’ve got that, you just need to run docker-compose up

This gives you a tiny one-broker Kafka cluster running locally with ephemeral storage.

I’ve included port-forwarding, so Kafka app running on your host machine can access the cluster at localhost:9092.

For example: --bootstrap-server localhost:9092 --topic MY.TOPIC --create --bootstrap-server localhost:9092 --topic MY.TOPIC --bootstrap-server localhost:9092 --topic MY.TOPIC

Or if you want to do everything from Docker containers, you can access it from other containers in the Docker Compose network at kafka:29092.

For example:

docker-compose exec kafka /bin/bash -c \
    "./bin/ --bootstrap-server kafka:29092 --topic MY.TOPIC --create"

docker-compose exec kafka /bin/bash -c \
    "./bin/ --bootstrap-server kafka:29092 --topic MY.TOPIC"

docker-compose exec kafka /bin/bash -c \
    "./bin/ --bootstrap-server kafka:29092 --topic MY.TOPIC"

A couple of caveats:

1) Although this is using the same images and compiled/built code as IBM Event Streams, running the images in this way skips a lot of the configuration and setup that happens in a real Event Streams deployment (such as setting up auth and encryption, enabling metrics, attaching additional agents, etc.). Running this Docker Compose setup does not get you a hardened, secured Kafka cluster. It’s a tiny single-broker unsecured cluster for development use.

2) This does run on Apple Silicon, thanks to the wonders of emulation. (That’s why I added the platform attributes.) There is a delay on some operations, like creating new topics. But once I had things up and running, I didn’t notice any problems.

Tags: , , ,

Comments are closed.