It’s Thanksgiving, so time to reflect on what I’m thankful for.
Okay, I’m not American… but I work for an American company and have a lot of American colleagues, so this week has been quieter than usual – that helped put me in a thankful mood! I thought I’d share some of the stuff I got this year that I have loved more than I expected.
In this post, I’ll walk through a sample implementation of using a database to maintain an Event Projection. I’ll use this to illustrate when this is a suitable approach to use.
In Comparing approaches to maintaining an Event Projection from Kafka topics, I introduced the pattern of Event Projections.
I also introduced the scenario that I’ll be using in these demos. Please see that post for the detail and motivation, but to recap, I want to maintain a projection of two Kafka topics (one based on the event key, the other based on an attribute in the event payload).
In both cases, I want to be able to make an HTTP/REST call to retrieve the data from the most recent event that matches my query.
At a high-level, the goal was to:
In this post, I’ll walk through a sample implementation of the simplest way to maintain an Event Projection: an in-memory lookup table. I’ll use this sample to illustrate when this is a suitable approach to use.
In Comparing approaches to maintaining an Event Projection from Kafka topics, I introduced the pattern of Event Projections.
I also introduced the scenario that I’ll be using in these demos. Please see that post for the detail and motivation, but to recap: I will maintain a projection of the data from two Kafka topics (one based on the event key, the other based on an attribute in the event payload).
In both cases, I want to be able to make an HTTP/REST call to retrieve the data that was in the most recent event to match my query.
At a high-level, the goal was to create a single server that will:
For demo purposes, my “application” will be curl, so I can illustrate being able to query the projection like this.
This is the first in a series of posts exploring different approaches to implementing the Event Projections pattern with Apache Kafka.
In this first post, I’ll introduce what Event Projections are, and outline some of the benefits of the Event Projections pattern.
Finally, I’ll introduce the scenario that I’ll be using to illustrate the pros and cons of different approaches in later posts.
Screen recording at youtu.be/cpCeaR9KTF8
MobileNet is a light-weight machine learning model for performing image classification.
In this Machine Learning for Kids project, students can try MobileNet for themselves using the familiar educational low-code programming language Scratch.
TLDR: I’ve been getting healthier this year. It’s gone well, and I’m pleased… so pleased that I feel the need to look back at what I’ve done.
In this post, I’ll show how to bring posts from open social media networks (Bluesky and Mastodon) into Kafka using Kafka Connect source connectors.
My goal is to be able to populate a Kafka topic with status updates posted to social media.
Rather than to try and do this with the full firehose of all status updates, this is done with status updates that match a search term or hashtag.
For example, the screenshot above is a Kafka topic with posts from Bluesky that mention the term “xbox”.
In this post, I’ll demonstrate how Event Processing can use parameters from an external source (such as a rules engine) in event processing flows.
To illustrate the idea, I created a simple demo event processing flow. The flow takes a stream of order events, filters it to keep only orders for high value items, and then modifies the description property in some of the events:
The filter node is comparing the price with “40”, so only order events for items with a value above $40 are kept.
The transform node is modifying the description property of order events – any description that contains the string “Cargo Jeans” is replaced with “Combat Trousers”.
What if you wanted to modify the threshold for the filter, to change that $40 minimum value for an order to be considered “large”?
Or what if you wanted to modify the transformation, so that different strings would be used in the regular expression replacement?
With the values hard-coded in the flow as shown above, you would need to:
This is a workable approach, although it does require a little downtime and some administrative effort.
The aim for this post is to highlight an alternative approach.