Archive for February, 2020

Bringing AI into the classroom

Friday, February 28th, 2020

IBM and mindSpark are running a series of free webinars for teachers about artificial intelligence.

This evening’s 90 minute webinar was about bringing AI into the classroom, and I helped contribute some of the content.

The session was very interactive, but there were some pre-prepared presentations in there.

I’ve got a recording of one of the segments below, in which I shared some of my experiences of introducing AI and machine learning in schools, and what I’ve found works well.

(more…)

Why are Kafka messages still on the topic after the retention time has expired?

Sunday, February 9th, 2020

We had an interesting Kafka question from an Event Streams user. The answer isn’t immediately obvious unless you know a bit about Kafka internals, and after a little searching I couldn’t find an explanation online, so I thought I’d share the answer here (obviously anonymised and heavily simplified).

What is retention?

Retention is a Kafka feature to help you manage the amount of disk space your topics use.

It lets you specify how long you want Kafka to keep messages on a topic for. You can specify this by time (e.g. “I want messages on this topic to be preserved for at least X days”) or by disk usage (e.g. “I want at least the last X gb of messages on this topic to be preserved”).

After the retention time or disk threshold is exceeded, messages become eligible for being automatically deleted by Kafka.

What was wrong in this case?

screenshot

They had created a topic with a retention time of 7 days.

They had assumed that this meant messages older than 7 days would be deleted.

When they looked at the messages on their topic, they could see some messages older than 7 days were there, and were surprised.

They thought this might mean retention wasn’t working.

(more…)