dale lane

In this post, I’ll walk through a sample implementation of using a database to maintain an Event Projection. I’ll use this to illustrate when this is a suitable approach to use.

The objective for this demo

In Comparing approaches to maintaining an Event Projection from Kafka topics, I introduced the pattern of Event Projections.

I also introduced the scenario that I’ll be using in these demos. Please see that post for the detail and motivation, but to recap, I want to maintain a projection of two Kafka topics (one based on the event key, the other based on an attribute in the event payload).

In both cases, I want to be able to make an HTTP/REST call to retrieve the data from the most recent event that matches my query.

At a high-level, the goal was to:

• use Kafka Connect JDBC sink connectors to maintain a database projection of the Kafka topics
• provide an HTTP/REST API for querying the projection

(more…)

In this post, I’ll walk through a sample implementation of the simplest way to maintain an Event Projection: an in-memory lookup table. I’ll use this sample to illustrate when this is a suitable approach to use.

The objective for this demo

In Comparing approaches to maintaining an Event Projection from Kafka topics, I introduced the pattern of Event Projections.

I also introduced the scenario that I’ll be using in these demos. Please see that post for the detail and motivation, but to recap: I will maintain a projection of the data from two Kafka topics (one based on the event key, the other based on an attribute in the event payload).

In both cases, I want to be able to make an HTTP/REST call to retrieve the data that was in the most recent event to match my query.

At a high-level, the goal was to create a single server that will:

• subscribe to my Kafka topics
• maintain an in-memory lookup of the relevant data
• provide an HTTP/REST API for querying the projection

For demo purposes, my “application” will be curl, so I can illustrate being able to query the projection like this.

(more…)

This is the first in a series of posts exploring different approaches to implementing the Event Projections pattern with Apache Kafka.

In this first post, I’ll introduce what Event Projections are, and outline some of the benefits of the Event Projections pattern.

Finally, I’ll introduce the scenario that I’ll be using to illustrate the pros and cons of different approaches in later posts.

(more…)