dale lane

I gave a talk at Current last week about how artificial intelligence and machine learning are used with Kafka topics. I had a lot of examples to share, so I wrote up my slides across several posts.

I’ll use this post to recap and link to my write-ups of each bit of the talk:

I started by talking about the different building blocks that are needed, and the sorts of choices that teams make.

Next, I talked about how projects to introduce AI into event driven architectures typically fall into one or more of these common patterns:

The most common, and the simplest: using AI to improve and augment the sorts of processing we can do with events. This can be as simple as using off-the-shelf pre-trained models to enrich a stream of events, and using this to filter or route the event as part of processing.

Perhaps the newest (and the pattern that is recently getting the most interest and attention) is to use streams of events to trigger agents, so that they can autonomously take actions in repsonse.

Maybe the less obvious approach is to collect and store a projection of recent events, and use these to enhance an agentic AI, by making it available as a queryable or searchable form of real-time context.

And finally, the longest established pattern is to simply use the retained history of Kafka topics as a relevant source of historical training data, for training new custom and bespoke models.

In this post, I describe how event streams can be used as a source of training data for machine learning models.

I spoke at Current last week. I gave a talk about how artificial intelligence and machine learning are most commonly used with Kafka topics. I had a lot to say, so I didn’t manage to finish writing up my slides – but this post covers the last section of the talk.

It follows:

• the building blocks used in AI/ML Kafka projects
• how AI / ML is used to augment event stream processing
• how agentic AI is used to respond autonomously to events
• how events can provide real-time context to agents
• how events can be used as a source of training data for models (this post)

The talk covered the four main patterns for using AI/ML with events.

This pattern was where I talked about using events as a source of training data for models. This is perhaps the simplest and longest established approach – I’ve been writing about this for years, long pre-dating the current generative AI-inspired interest.

(more…)

In this post, I describe how agentic AI can respond autonomously to event streams.

I spoke at Current on Wednesday, about the most common patterns for how AI and ML are used with Kafka topics. I had a lot of content I wanted to cover in the session, so it’s taking me a while to write it all down:

• the building blocks used in AI/ML Kafka projects
• how AI / ML is used to augment event stream processing
• how agentic AI is used to respond autonomously to events (this post)
• how events can provide real-time context to agents
• how events can be used as a source of training data for models

The premise of the talk was to describe the four main patterns for using AI/ML with events. This pattern was where I started focusing on agents.

(more…)

In this post, I describe how artificial intelligence and machine learning are used to augment event stream processing.

I gave a talk at a Kafka / Flink conference yesterday about the four main patterns for using AI/ML with events. I had a lot to say, so it is taking me a few days to write up my slides.

• the building blocks used in AI/ML Kafka projects
• how AI / ML is used to augment event stream processing (this post)
• how agentic AI is used to respond autonomously to events
• how events can provide real-time context to agents
• how events can be used as a source of training data for models

The most common pattern for introducing AI into an event driven architecture is to use it to enhance event processing.

As part of event processing, you can have events, collections of events, or changes in events – and any of these can be sent to an AI service. The results can inform the processing or downstream workflows.

(more…)