{"id":5505,"date":"2025-02-03T20:58:20","date_gmt":"2025-02-03T20:58:20","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5505"},"modified":"2025-02-03T20:58:20","modified_gmt":"2025-02-03T20:58:20","slug":"running-openmessaging-benchmarks-on-your-kafka-cluster","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5505","title":{"rendered":"Running OpenMessaging benchmarks on your Kafka cluster"},"content":{"rendered":"

The OpenMessaging Benchmark Framework<\/a> is typically used to benchmark messaging systems in the cloud, but in this post I want to show how useful it can also be for Kafka clusters that you run yourself in Kubernetes (whether that is using the open source Strimzi operator, or IBM’s Event Streams).<\/strong><\/p>\n

From openmessaging.cloud<\/a>:<\/p>\n

The OpenMessaging Benchmark Framework is a suite of tools that make it easy to benchmark distributed messaging systems.<\/p><\/blockquote>\n

As I’ve written about before (when illustrating the impact of setting quotas at the Kafka cluster level<\/a>, and when adding quotas at the event gateway level<\/a>), Apache Kafka comes with a good performance test tool. That is still my go-to option if I just want an easy way to push data through a Kafka cluster in bulk.<\/p>\n

But – OpenMessaging’s benchmark has some interesting features that make it a useful complement and worth considering.<\/p>\n

The benefit that OpenMessaging talk about the most is that can be used with a variety of messaging systems, such as RocketMQ, Pulsar, RabbitMQ, NATS, Redis and more – although in this post I’m only interested in using it to benchmark an Apache Kafka cluster.<\/p>\n

More interesting for me was their focus on realistic workloads<\/strong> rather than relying on static data.<\/p>\n

Quoting again from openmessaging.cloud<\/a>:<\/p>\n

Benchmarks should be largely oriented toward standard use cases rather than bizarre edge cases<\/p><\/blockquote>\n

I also like the way that the framework orchestrates running distributed benchmarks<\/strong>. It deploys a “driver” pod and multiple “worker” pods into your Kubernetes cluster.<\/p>\n

You open a terminal into the driver pod, and start a benchmark by running a command-line command. The driver pod makes HTTP calls to each of worker pods to get them all to participate in a distributed benchmark workload, and it collects the results from each worker into a single collated file for you.<\/p>\n

It also starts each benchmark by getting all of the worker pods to participate in a warmup phase, before it starts the actual measured benchmark.<\/p>\n

This all works well and it is a very simple system.<\/p>\n

Specifying a benchmark<\/h3>\n

The command you run is just:<\/p>\n

benchmark \\\n    --workers $WORKERS \\\n    --drivers DRIVER-NAME-HERE \\\n    WORKLOAD-NAME-HERE<\/pre>\n
The workers flag is where you provide the location of your worker pods, but the driver pod has the environment variable WORKERS<\/code> automatically configured for you, so that bit is easy.<\/p>\n
The only decisions you need to make are what driver<\/strong> and workload<\/strong> to use for your benchmark. Each one is defined as a simple, human-readable YAML file containing standard Kafka client configuration properties<\/a> so you can add your own if you want.<\/p>\n
Choosing a driver<\/h3>\n
The driver<\/strong> option is where you specify how you want the Kafka clients used in your benchmark to behave<\/strong>.<\/p>\n
For example, do you want the clients optimised for high throughput (kafka-throughput.yaml<\/code><\/a>) or low latency (kafka-latency.yaml<\/code><\/a>)?<\/p>\n
Do you want clients to use a compression algorithm (e.g. kafka-compression-gzip.yaml<\/code><\/a>, kafka-compression-zstd.yaml<\/code><\/a>, kafka-compression-lz4.yaml<\/code><\/a>)?<\/p>\n
Do you want the clients to wait for and produce large batches (e.g. kafka-big-batches.yaml<\/code><\/a>) or send events immediately without waiting (kafka-no-linger.yaml<\/code><\/a>)?<\/p>\n
Do you want the clients to optimise for preventing message duplication due to network retries (kafka-exactly-once.yaml<\/code><\/a>)?<\/p>\n
And so on – you can see the drivers available for Kafka benchmarks in the driver-kafka<\/code> folder<\/a>.<\/p>\n
Choosing a workload<\/h3>\n
The workload<\/strong> option is where you specify what you want the Kafka clients to do<\/strong>.<\/p>\n
For example, how many topics do you want it to use? How many partitions should the topics have? How many consumers and producers do you want to run? Do you want the producers to run at a fixed rate, or to produce as fast as possible? What size messages do you want the producers to create? Do you want to produce fixed static data, or to generate random data? How long do you want the benchmark to run for? There are many choices!<\/p>\n
Each workload is defined as a human-readable YAML file – try clicking on one of the example workloads in the scrollable list below to see what they look like:<\/p>\n