{"id":6001,"date":"2026-04-11T16:28:05","date_gmt":"2026-04-11T16:28:05","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=6001"},"modified":"2026-04-13T19:49:51","modified_gmt":"2026-04-13T19:49:51","slug":"how-many-kafka-events-will-flink-process-per-second","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=6001","title":{"rendered":"“How many Kafka events will Flink process per second?”"},"content":{"rendered":"

\nI’m often asked this. The specific question varies, but it’s typically some variation of asking how quickly a single CPU of Flink processes events from a Kafka topic.\n<\/p>\n

\n

\nWhy “per CPU”? Maybe because enterprise software is typically charged per CPU? Maybe because I tend to talk to people who run everything in Kubernetes, who think of running software in terms of requests \/ limits<\/a>? Not sure, but the question tends to be framed from the perspective of asking how much processing they can expect to get from a CPU.<\/em>\n<\/p>\n<\/blockquote>\n

\nI try to avoid doing the engineer thing of answering “it depends<\/a>“… but… it really does<\/strong> depend!\n<\/p>\n

\n<\/a>\n<\/p>\n

\nThat is the motivation behind this post: to give me something I can point at as an illustration of the degree to which Flink’s performance varies (and a taste of the range of interrelated factors that influence it).\n<\/p>\n

<\/p>\n

How different can it be?<\/h3>\n

\nI’ve talked before that my go-to example when demo’ing Flink SQL for now are Flink SQL flows that process click tracking events in a retail scenario<\/a>. They are easy to explain, easy to understand, and let me show a variety of Flink SQL features.\n<\/p>\n

\nUsing that Kafka topic, and those sorts of flows as a basis, how fast do the Flink jobs I demo consume events (if I set the CPU limit to 1 for the job manager and task manager)?\n<\/p>\n

\nI ran those flows a bunch of times, changing some configuration options<\/a> along the way, to see what sort of range of speeds I could get as a result. When I plot each Flink job onto an x-axis based on the number of records it processed per second, I get:\n<\/p>\n

\n\n<\/div>\n

\nLook at the range there.\n<\/p>\n

\nAt one end, one of the flows consumed approximately 1,340 events per second.<\/p>\n

\nAt the other end, another job consumed approximately 44,320 events per second.\n<\/p>\n

\n

\nTLDR<\/strong> – If you takeaway nothing else from this post, that’s the message. All of those flows were Flink running in the same place, consuming the same type of data, from the same Kafka cluster, using the same amount of CPU and memory.\n<\/p>\n

\nThe throughput varied massively depending on the type of processing the flow was doing, the type of data it was consuming, and how Flink was configured<\/strong>.\n<\/p>\n<\/blockquote>\n

\nThese results are specific to my environment, workload, and the configurations I tried. (And I took some questionable shortcuts that I describe below!) I’m sharing this as an interesting illustration of how variable Flink performance can be, and as an encouragement of the value of testing and benchmarking.\n<\/p>\n

\nIn other words, I guess I’m still saying it depends<\/a>. \ud83d\ude09\n<\/p>\n

Digging into the differences<\/h3>\n

\nIf you hover over the points in the visualisation above, the tooltip gives an overview of what I did in that Flink job.\n<\/p>\n

\nSeeing all of them in one group is helpful to get a feel for the distribution, but it doesn’t make it easy to see patterns. While I’ve got this data, it’s worth poking at it a little closer.\n<\/p>\n

\nFor example, let me pick out two of the Flink job runs. These two Flink jobs both:<\/p>\n