{"id":5560,"date":"2025-05-05T11:52:10","date_gmt":"2025-05-05T11:52:10","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5560"},"modified":"2025-05-05T11:52:40","modified_gmt":"2025-05-05T11:52:40","slug":"visualising-apache-kafka-events-in-grafana","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5560","title":{"rendered":"Visualising Apache Kafka events in Grafana"},"content":{"rendered":"<p><strong>In this post, I want to share some ideas for how Grafana could be used to create visualisations of the contents of events on Apache Kafka topics.<\/strong><\/p>\n<p>By using Kafka as a data source in Grafana, we can create dashboards to query, visualise, and explore live streams of Kafka events. I\u2019ve recorded <a href=\"https:\/\/youtu.be\/EX5clcmHRsU\">a video where I play around with this idea<\/a>, creating a variety of different types of visualisation to show the sorts of things that are possible. <\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"280\" src=\"https:\/\/www.youtube.com\/embed\/EX5clcmHRsU?si=tC4UMfH7D8QV_w5X\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><br \/>\n<small><a href=\"https:\/\/youtu.be\/EX5clcmHRsU\">youtu.be\/EX5clcmHRsU<\/a><\/small><\/p>\n<p>To make it easy to skim through the examples I created during this run-through, I\u2019ll also share screenshots of each one below, with a time-stamped link to the part of the video where I created that example. <\/p>\n<p>Finally, at the end of this post, I\u2019ll talk about the mechanics and practicalities of how I did this, and what I think is needed next. <\/p>\n<p><!--more--><\/p>\n<hr \/>\n<h3>Adding a local, insecure Kafka cluster as a data source<\/h3>\n<p>Not a visualisation, but a necessary first step. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/01-data-source-unauth.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/01-data-source-unauth.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=66\">starts at 01:06 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Visualising values from sensor readings<\/h3>\n<p>The topic I used for the first few examples was a topic with data from temperature and humidity sensors.<\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/00-sensors-topic.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/00-sensors-topic.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=35\">starts at 00:35 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Display a stream of Kafka events in a live, updating table<\/h3>\n<p>A useful start before jumping into graphics and visualisations is to just display the JSON events in a tabular form. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/02-data-table.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/02-data-table.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=100\">starts at 01:40 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Show values from the most recent event in a gauge<\/h3>\n<p>A live, dynamic display of the values from the most recent event on the topic, using a curved bar and dynamic colours to indicate what the values mean.<\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/03-gauge.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/03-gauge.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=191\">starts at 03:11 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Generate a time series line for recent numerical data in events<\/h3>\n<p>A live, updating line graph can show how the values in the events change over time. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/04-timeseries.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/04-timeseries.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=408\">starts at 06:48 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Visualising values from order events<\/h3>\n<p>The topic I used for the next examples was an orders topic with events containing a mixture of numeric, string, and categorical values. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/05-orders-topic.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/05-orders-topic.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=582\">starts at 09:42 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Display a stream of Kafka events in a live, updating table<\/h3>\n<p>As before, I started with a scrollable, paginated view of the JSON events in a table. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/06-data-table.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/06-data-table.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=628\">starts at 10:28 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Using a bar chart to understand the distribution of categorical data<\/h3>\n<p>A dynamic bar chart that shows the number of events that included different values for one of the enums \/ categorical properties in the data.<\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/07-bar-chart-distribution.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/07-bar-chart-distribution.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=687\">starts at 11:27 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Using a histogram to understand the distribution of numerical data<\/h3>\n<p>A dynamic histogram that shows the number of events with numerical values within certain ranges.  <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/08-histogram-distribution.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/08-histogram-distribution.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=861\">starts at 14:21 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Using a bar chart to display the frequency of events<\/h3>\n<p>It isn\u2019t just the contents of the events that we can visualise, but also metadata such as how frequently events are being produced. I used a bar chart to show the number of order events per minute.<\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/09-bar-chart-counts.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/09-bar-chart-counts.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=1008\">starts at 16:48 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Displaying statistical summaries of numerical data<\/h3>\n<p>A dynamic table showing simple statistical summaries of numerical values in the events. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/10-summary-table.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/10-summary-table.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=1270\">starts at 21:10 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Adding a remote, authenticated Kafka cluster as a data source<\/h3>\n<p>Not a visualisation, but a necessary first step. <\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/11-data-source-auth.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/11-data-source-auth.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=1408\">starts at 23:28 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Plotting geo-spatial events on a map<\/h3>\n<p>A stream of events that contain location data can be visualised as live, moving labelled events on a dynamic map &#8211; displaying additional data about map points when you hover over them.<\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/12-geomap.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/12-geomap.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=1517\">starts at 25:17 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Displaying distribution of events across topic partitions in a pie chart<\/h3>\n<p>It isn\u2019t just the contents of the events that we can visualise, but also metadata such as the distribution of events on a multi-partition topic. This could help show the effectiveness of the key strategy in the data.<\/p>\n<p><a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/blob\/master\/screenshots\/13-pie-chart.png\"><img decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/dalelane\/grafana-kafka-datasource\/refs\/heads\/master\/screenshots\/13-pie-chart.png\" border=0 style=\"width: 100%; max-width: 900px;\"\/><\/a><\/p>\n<p><a href=\"https:\/\/youtu.be\/EX5clcmHRsU?t=1834\">starts at 30:34 in the video<\/a><\/p>\n<p><\/p>\n<hr \/>\n<h3>Other visualisations<\/h3>\n<p>Those were just what I managed to come up with during half-an-hour of experimentation and playing. I\u2019m sure that someone with more imagination and Grafana skills could do more!<\/p>\n<h3>Mechanics and practicalities<\/h3>\n<p>Finally, I\u2019ll finish by talking about what we need to be able to do this sort of thing. Grafana\u2019s support for different data sources comes from data source plugins &#8211; so we need a Grafana data source plugin that can run Kafka consumers. <\/p>\n<p>I couldn\u2019t find one, so I had a go at creating it &#8211; you can find it at <a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\">github.com\/dalelane\/grafana-kafka-datasource<\/a>.<\/p>\n<p>This is not only my first exploration of <a href=\"https:\/\/grafana.com\/developers\/plugin-tools\/tutorials\/build-a-streaming-data-source-plugin\">working with the Grafana API<\/a>, it\u2019s also my first attempt to write anything using the Go programming language (<em>part of the motivation for this project was giving me an excuse to give Golang a try, which I\u2019ve meaning to do for years!<\/em>). <\/p>\n<p>All of this is a long-winded excuse to say that my plugin is still a little rough around the edges but it\u2019s a start. (I would welcome input from people who actually know what they\u2019re doing in Go to point out the more ignorant and naive aspects of what I\u2019ve done!) <\/p>\n<p>I\u2019ve submitted it to Grafana so I can get it signed. That will make it easier for you to install it in your Grafana instance and try it out for yourself. (I\u2019ve not done this before, so I have no idea how long <a href=\"https:\/\/grafana.com\/developers\/plugin-tools\/publish-a-plugin\/publish-a-plugin\">this process<\/a> takes.)  <\/p>\n<p>In the meantime, you can still install <a href=\"https:\/\/github.com\/dalelane\/grafana-kafka-datasource\/releases\">the unsigned plugin from my Github repo<\/a>, but you\u2019ll need to <a href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/administration\/plugin-management\/#allow-unsigned-plugins\">allow unsigned plugins<\/a> for this to work.<\/p>\n<h3>What now?<\/h3>\n<p>In summary, I\u2019ve got a plugin that mostly works, and (<em>at time of writing in early May 2025<\/em>) is perhaps best thought of as a pre-release beta. I thought it was worth sharing even in this early phase to see what interest there is. <\/p>\n<p>Please get in touch if:<\/p>\n<ul>\n<li>you think you\u2019d find this useful<\/li>\n<li>you know Go and can see what I\u2019ve done wrong<\/li>\n<li>you know Go and would like to contribute to the plugin<\/li>\n<li>you have ideas for what else could be done using Kafka as a Grafana data source<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>In this post, I want to share some ideas for how Grafana could be used to create visualisations of the contents of events on Apache Kafka topics. By using Kafka as a data source in Grafana, we can create dashboards to query, visualise, and explore live streams of Kafka events. I\u2019ve recorded a video where [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":5565,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,584],"class_list":["post-5560","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-apachekafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5560","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5560"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5560\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/5565"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5560"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5560"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5560"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}