This is the first in a series of blog posts sharing examples of ways to use Mirror Maker 2 with IBM Event Streams.
- Using Mirror Maker 2 to aggregate events from multiple regions
- Using Mirror Maker 2 to broadcast events to multiple regions
- Using Mirror Maker 2 to share topics across multiple regions
- Using Mirror Maker 2 to create a failover cluster
- Using Mirror Maker 2 to restore events from a backup cluster
- Using Mirror Maker 2 to migrate to a different region
Mirror Maker 2 is a powerful and flexible tool for moving Kafka events between Kafka clusters, but sometimes I feel like this can be forgotten if we only talk about it in the context of disaster recovery.
In these posts, I want to inspire you to think about other ways you could use Mirror Maker 2. The best way to learn about what is possible is to play with it for yourself, so with these posts I’ll include a script to create a demonstration of the scenario.
For this first post, I’ll look at using Mirror Maker to aggregate events across clusters in multiple regions.
Imagine that you are producing events from applications running in multiple regions, and you need to consume and process them all together.
If you do this with a single Kafka cluster, there are inevitable compromises.
You could locate each Kafka topic with the application that is processing the events, but this introduces latency for the producers.
You could improve this by locating the Kafka topics with the producing applications.
This improves the situation for the producers, but it isn’t optimal for a stream processing application to consume events from multiple remote locations.
Mirroring is a way of getting the best of both worlds for applications in all regions.
Applications in all regions get the benefits of interacting with a local Kafka cluster. Mirror Maker takes the responsibility of aggregating events into a single place ready for the stream processing app. In addition to latency benefits, aggregating them into a single Kafka cluster enables the stream processing app to use transactions across the topics it consumes from, which is useful in enabling different types of processing.
Using Mirror Maker 2 is a good fit where events are being generated in multiple locations, and need to be processed together.
Demo
For a demonstration of this, I created:
Three Kubernetes namespaces (“north-america”, “south-america”, “europe”) represent three different regions. An Event Streams Kafka cluster is created in each “region”.
A producer application is started in each of the “North America” and “South America” regions. These producers regularly produce randomly generated events, themed around a fictional clothing retailer, Loosehanger Jeans.
The application in the “North America region” generates random orders and warehouse stock movement events with North American locations. The application in the “South America region” generates random orders and warehouse events with South American locations.
A Mirror Maker instance running in the “Europe region” aggregates all of these to topics in the Event Streams cluster there. The destination topics in the Europe region are prefixed with the source cluster name to allow them to be distinguished.
A single consuming application subscribes to the combination of all of these topics, receiving all of the events from both American regions.
To create the demo for yourself
There is an Ansible playbook here which creates all of this:
github.com/dalelane/eventstreams-mirrormaker2-demos/blob/master/01-aggregate-central/setup.yaml
An example of how to run it can be found in the script at:
setup-01-aggregate-central.sh
This script will also display the URL and username/password for the Event Streams web UI for all three regions, to make it easier to log in and see the events.
Once you’ve created the demo, you can run the consumer-europe.sh script to see the events that are received by the consumer application in the “Europe region”.
(Once you’ve finished, the cleanup.sh script deletes everything that the demo created.)
How the demo is configured
The Mirror Maker config can be found here: mm2.yaml.
The spec is commented so that is the main file to read if you want to see how to configure Mirror Maker to satisfy this kind of scenario.
An alternative topology
Notice that I ran the Mirror Maker 2 pod in the “Europe region”. The optimum place to locate Mirror Maker is alongside the target cluster, so that the producer end has the best connection.
However, you may not have the option to add new runtime components in the target region. In these situations, you could create this instead:
The result is the same. Events from producer applications in both the “North America region” and “South America region” are being aggregated into a Kafka cluster in the “Europe region” – where a single consumer application is receiving all of them.
In this example, the work to aggregate the events is being done by two separate instances of Mirror Maker, each running alongside the source clusters.
To create this demo for yourself
There is an Ansible playbook here which creates all of this:
github.com/dalelane/eventstreams-mirrormaker2-demos/blob/master/02-aggregate-distributed/setup.yaml
An example of how to run it can be found in the script at: setup-02-aggregate-distributed.sh
(Note that you should clean up the previous demo first before running this, to avoid running three instances of a Mirror Maker pod!)
Once you’ve created the demo, you can again run the consumer-europe.sh script to see the events that are received by the consumer application in the “Europe region”.
(And the cleanup.sh script will similarly delete everything that the demo created.)
How this demo is configured
The Mirror Maker configs can be found here:
mm2-na.yaml(Mirror Maker in the “North American region”)mm2-sa.yaml(Mirror Maker in the “South American region”)
The specs are commented so these are the files to read if you want to see how to configure Mirror Maker to satisfy this kind of scenario.
I’ve only talked here about how MM2 is moving the events between regions, but if you look at the comments in the mm2 specs, you’ll see that it is doing more than that. For example, it is also keeping the topic configuration in sync. Try that for yourself. Modify the configuration of one of the topics in the “North American region” and then see that change reflected in the corresponding topic in the “Europe region”.
More scenarios to come
This is one of the simplest ways to use Mirror Maker 2. In the next few posts, I’ll share other demos that show other scenarios.
Tags: apachekafka, ibmeventstreams, kafka