{"id":5483,"date":"2025-01-31T14:01:15","date_gmt":"2025-01-31T14:01:15","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5483"},"modified":"2026-03-14T21:28:42","modified_gmt":"2026-03-14T21:28:42","slug":"understanding-event-processing-behaviour-with-opentelemetry","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5483","title":{"rendered":"Understanding event processing behaviour with OpenTelemetry"},"content":{"rendered":"

When using Apache Kafka, timely processing of events is an important consideration.<\/p>\n

Understanding the throughput<\/strong> of your event processing solution is typically straightforward : by counting how many events you can process a second.<\/p>\n

Understanding latency<\/strong> (how long it takes from when an event is first emitted, to when it has finished being processed and an action has been taken in response) requires more coordination to be able to measure.<\/p>\n

OpenTelemetry<\/strong> helps with this, by collecting and correlating information from the different components that are producing and consuming events.<\/p>\n

From opentelemetry.io<\/a>:<\/p>\n

OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software\u2019s performance and behavior.<\/p><\/blockquote>\n

A distributed event processing solution<\/h3>\n

To understand what is possible, I\u2019ll use a simple (if contrived!) example of an event processing architecture.<\/p>\n

<\/p>\n

In this demo:<\/p>\n

    \n
  1. producer<\/code>
    A Java application<\/strong> reads JSON files in a folder, and produces the contents of each file as an event to a Kafka topic called OTEL.INPUT<\/code><\/li>\n
  2. enrich<\/code>
    A Flink SQL job<\/strong> consumes the events from OTEL.INPUT<\/code>, enriches each event with additional data it retrieves from a REST API, and produces the output to a Kafka topic called OTEL.PROCESSING<\/code><\/li>\n
  3. filter<\/code>
    A Kafka Streams<\/strong> application consumes events from the OTEL.PROCESSING<\/code> topic, applies a filter and produces matching events to a Kafka topic called OTEL.OUTPUT<\/code><\/li>\n
  4. consumer<\/code>
    A Java application<\/strong> consumes events from the OTEL.OUTPUT<\/code> Kafka topic and prints the contents to the console<\/li>\n<\/ol>\n

    Understanding the behaviour<\/h3>\n

    A simple understanding of the latency for this solution means understanding how long it takes from when producer<\/code> sends the file contents to the OTEL.INPUT<\/code> topic, to when consumer<\/code> prints out the results from processing it.<\/p>\n

    A detailed insight into the performance of this solution means understanding the time spent by each of the parts of the solution.<\/p>\n

    Instrumenting each of the components lets them independently submit \u201cspans\u201d (recordings of what they did and when) to an OpenTelemetry collector. The collector correlates the spans it receives into end-to-end traces.<\/p>\n

    <\/p>\n

    Deploying the OpenTelemetry collector<\/h3>\n

    OpenTelemetry is the specification and protocol for how to do all of this, and there are several implementations available.<\/p>\n

    For my demo, I used Grafana Tempo, because there is a Kubernetes Operator that made it very easy to create a quick demo.<\/p>\n

    apiVersion: tempo.grafana.com\/v1alpha1\nkind: TempoMonolithic\nmetadata:\n  name: otel\n  namespace: monitoring\nspec:\n  jaegerui:\n    enabled: true\n    resources:\n      limits:\n        cpu: '2'\n        memory: 2Gi\n    route:\n      enabled: true\n  resources:\n    limits:\n      cpu: '2'\n      memory: 2Gi\n  storage:\n    traces:\n      backend: memory<\/pre>\n
    01-otel-collector.yaml<\/code><\/a><\/div>\n
    There are many alternative OpenTelemetry implementations that I could have used instead. You can see a list on the OpenTelemetry website at opentelemetry.io\/ecosystem\/vendors<\/a><\/p>\n
    Instrumenting the producer<\/h3>\n
    <\/p>\n
    The producer<\/code> application uses the Kafka Java client to send events to Kafka.<\/p>\n
    No code changes were needed to get it to submit trace spans to the OpenTelemetry collector, but a few config changes were needed. You can see them in this commit<\/a>, but I’ll walk through them here.<\/p>\n
    The simplest way to add the additional Java libraries<\/strong> needed was to get Maven to fetch them for me, by adding this to the dependencies list in my pom.xml<\/p>\n
    \n<dependency>\n    <groupId>io.opentelemetry.instrumentation<\/groupId>\n    <artifactId>opentelemetry-kafka-clients-2.6<\/artifactId>\n    <version>2.11.0-alpha<\/version>\n<\/dependency>\n<\/pre>\n
    pom.xml<\/code><\/a> (commit)<\/a><\/div>\n
    I updated the properties file<\/strong> that is used to configure the Kafka Producer by adding this line:<\/p>\n
    \ninterceptor.classes=io.opentelemetry.instrumentation.kafkaclients.v2_6.TracingProducerInterceptor\n<\/pre>\n
    producer.properties<\/code><\/a> (commit)<\/a><\/div>\n
    Finally I updated the script used to run the application to add a Java agent and environment variables<\/strong> that configure:<\/p>\n