dale lane

We had an interesting Kafka question from an Event Streams user. The answer isn’t immediately obvious unless you know a bit about Kafka internals, and after a little searching I couldn’t find an explanation online, so I thought I’d share the answer here (obviously anonymised and heavily simplified).

What is retention?

Retention is a Kafka feature to help you manage the amount of disk space your topics use.

It lets you specify how long you want Kafka to keep messages on a topic for. You can specify this by time (e.g. “I want messages on this topic to be preserved for at least X days”) or by disk usage (e.g. “I want at least the last X gb of messages on this topic to be preserved”).

After the retention time or disk threshold is exceeded, messages become eligible for being automatically deleted by Kafka.

What was wrong in this case?

They had created a topic with a retention time of 7 days.

They had assumed that this meant messages older than 7 days would be deleted.

When they looked at the messages on their topic, they could see some messages older than 7 days were there, and were surprised.

They thought this might mean retention wasn’t working.

What are segments?

To explain this, I need to give a bit of background about how Kafka stores messages on disk.

Instead of every message being in its own file, messages are collected together in “segment” files.

For example, in this diagram, my topic partition is shown in grey. Messages are shown as white boxes. And the physical segment file is shown in blue.

Click to enlarge

New messages that are produced to the topic partition are appended to the end of the active segment file.

Click to enlarge

When the broker decides the segment file is big enough, it starts a new one.

The new file becomes the active segment, and new messages will be appended to that file instead.

Click to enlarge

When will Kafka start a new segment file?

You can control this using some Kafka config parameters:

• The maximum age of messages in a segment file (segment.ms)
• The maximum size of a segment file (segment.bytes)

If either of these are exceeded, the broker will start a new segment.

For example:

max age

If segment.ms is set to 7 days, then the broker starts a new segment when the oldest message in a segment is older than 7 days.

Click to enlarge

In this case, the broker started segment 8 because the oldest message (offset 000) was older than 7 days old.

max size

If segment.bytes is set to 100mb, then the broker starts a new segment when the segment file reaches 100mb.

Click to enlarge

In this case, even though the oldest message is less then 7 days old, the broker still starts a new segment because the segment file has gotten too big.

What does segments have to do with retention?

Kafka retention doesn’t edit a segment file – it will only delete entire segment files. This is obviously much more performant.

If every message in a segment is eligible for deletion based on the retention rules, the whole segment file is deleted.

If even one message in a segment is not yet eligible for deletion based on retention, the broker will keep the entire segment file.

How did this user end up with messages older than 7 days on their topic?

Their topic has a retention.ms config value set to 7 days.

They had segment.ms (maximum age of a segment file) set to 7 days.
And they had segment.bytes set to 1gb.

Their topic was a bit like this (again, heavily simplified to make this easier to explain):

The oldest remaining segment – segment 100 – was started on 28th Jan.

The next segment – segment 107 – was started on 4th Feb, when the oldest message on segment 100 was older than the segment.ms limit.

There were a large number of messages produced on the next few days, so the next segment – segment 118 – was started on the 7th Feb, when segment 107 reached the segment.bytes limit.

Click to enlarge

Today is 9th Feb.

The oldest segment – segment 100 – still can’t be deleted by the 7-day topic retention policy, because the most recent message in that segment is only 5 days old.

The result is that they can see messages dating back to 28th January on this topic, even though the 28th Jan is now 12 days ago.

In a couple of days, all the messages in segment 100 will be old enough to exceed the retention threshold so that segment file will be deleted.

Click to enlarge

But by this point, such as on the 12th Feb, although the January messages will now be removed, they will see messages dating back to 4th Feb on their topic. This means they will still have messages older than 7 days on their topic.

To sum up

Don’t think of topic retention as a per-message guarantee of when a Kafka message will definitely be deleted.

It doesn’t work like that.

Instead, think of topic retention as a threshold that the Kafka broker can use to decide when messages are eligible for being automatically deleted. But the actual deletion might not happen for a while after that, to help the garbage collection process be more efficient.

Tags: apachekafka, ibmeventstreams, kafka

This entry was posted on Sunday, February 9th, 2020 at 8:43 pm and is filed under tech. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.