Written with Chris Patmore
IBM Event Automation helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have.
A key goal of Event Automation is to be composable. The three components can be used together, or they can each be used to extend and enhance an existing event-driven deployment.
Amazon MSK (Managed Streaming for Kafka) is a hosted, managed Kafka service available in Amazon Web Services. If a business has started their event-driven journey using MSK, then components from Event Automation can help to enhance this. This could be by offering management and governance of their MSK topics. And it could be by providing an intuitive low-code authoring canvas to process the events on their MSK topics.
Working with Amazon MSK is a nice example of the benefits of the composability of Event Automation, by helping businesses to get more value from their existing MSK topics.
In this blog post, we want to show a few different examples of where this can be done. For each example, we’ll provide a high-level diagram and description. We’ll also share a demonstration that we created to show it in action.
(Click on the descriptions for detailed step-by-step instructions and screenshots of how we built each demo.)
To start with, we demonstrated how Event Processing can be used with Amazon MSK. We showed how Event Processing, based on Apache Flink, can help businesses to identify insights from the events on their MSK topics through an easy-to-use authoring canvas.
The diagram above is a simplified description of what we created. We created an MSK cluster in AWS, set up a few topics, and then started a demonstration app producing a stream of events to them. This gave us a simplified example of a live MSK cluster.
We then accessed this Amazon MSK cluster from an instance of Event Processing (that was running in a Red Hat OpenShift cluster in IBM Cloud). We used Event Processing to create a range of stateful stream processing flows.
This showed how the events on MSK topics can be processed where they are, without requiring an instance of Event Streams or for topics to be mirrored into a Kafka cluster also running in OpenShift. Using the low-code authoring canvas with Kafka topics that you already have, wherever they are, is a fantastic event-driven architecture enabler.
How we created a demonstration of this…
We started by creating an Amazon MSK cluster.
▶
We opened the Amazon Web Services admin console, and went to the MSK (Managed Streaming for Apache Kafka) service.
To start, we clicked Create cluster.
We opted for the Custom create option so we could customize our MSK cluster.
We called the cluster loosehanger-msk because we’re basing this demonstration on “Loosehanger” – a fictional clothes retailer that we have a data generator for.
We chose a Provisioned (rather than serverless) Kafka cluster type, and chose the latest version of Kafka that Amazon offered (3.5.1).
Because we only needed an MSK cluster for this short demo, we went with the small broker type.
We went with the default, and recommended, number of zones to distribute the Kafka brokers across: three.
Because we only planned to run a few applications with a small number of topics, we didn’t need a lot of storage – we gave each broker 5GB of storage.
Rather than go with the default Kafka configuration, we clicked Create configuration to prepare a new custom config.
We gave the config a name similar to the MSK cluster itself, based on our scenario of the fictional clothes retailer, Loosehanger.
We started with a config that would make it easy for us to set up the topics.
auto.create.topics.enable=true default.replication.factor=3 min.insync.replicas=2 num.io.threads=8 num.network.threads=5 num.partitions=1 num.replica.fetchers=2 replica.lag.time.max.ms=30000 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 unclean.leader.election.enable=true zookeeper.session.timeout.ms=18000 allow.everyone.if.no.acl.found=true
The key value we added to the default config was the allow.everyone.if.no.acl.found one, to make it clear that we would start creating topics before setting up auth or access control lists.
We clicked Create to create this configuration. Once back on the MSK cluster settings screen, we chose this custom config and clicked Next to move on to the next step.
Networking was next.
We clicked Create VPC to prepare a new virtual networking environment for our demonstration.
We opted for the VPC and more option so we could set this up in a way that would support public access to the MSK cluster.
We chose three availability zones to match the config we used for the MSK cluster – this would allow us to have a separate AZ for each MSK broker.
We went with three public subnets and no private subnets, again as this would allow us to enable public access to the MSK cluster.
We left the default DNS options enabled so that we could have DNS hostnames for our addresses.
Next we clicked Create VPC.
We verified that the VPC resources we requested were created and then closed this window.
Back on the Networking step of the MSK cluster creation wizard, we were now able to choose our new VPC, and select the zones and subnets. The match of three availability zones for the MSK cluster, and three availability zones for the VPC meant it was just a matter of choosing a different zone for each broker.
We wanted to enable public access, but this can’t be done at cluster creation time, so this remained on Off for now.
The final networking step option is to create security groups. The default option here was fine for our purposes, so we left this as-is.
We clicked Next to move onto the next step.
The next step was to configure the security options for the MSK cluster.
We started with disabling auth, as unauthenticated access would make it easy for us to set up our cluster and topics. We enabled auth after we had the cluster the way we wanted it.
For the same reason, we also left client TLS disabled as well. We turned this on later when we enabled public access to the cluster.
The default encryption key was fine for encrypting the broker storage, so we left this as-is and clicked Next.
The next step is to configure monitoring. As a short-lived demo cluster, we didn’t have monitoring requirements, so we left this on the default basic option and clicked Next.
Our MSK cluster specification was ready to go, so we clicked Create.
We had to wait for the MSK cluster to provision.
Once it was ready, we could move on to the next stage which was to create topics.
Next, we created some Kafka topics that we would use with Event Automation, and set up some credentials for accessing them.
▶
Amazon MSK doesn’t offer admin controls for the Kafka cluster, so we needed somewhere that we could run a Kafka admin client.
The simplest option was to create an EC2 server where we could run Kafka admin commands from. We went to the EC2 service within AWS, and clicked Launch instance.
We kept with our naming theme of “Loosehanger” to represent our fictional clothes retailer.
This would only be a short-lived server, that we would keep around long enough to run a few Kafka admin commands, so quick and simple was the priority. We went with the Amazon Linux quick start option.
Again, as this would only be a short-lived server running a small number of command-line scripts, a small, free-tier-eligible instance type was fit for our needs.
We didn’t need to create a key pair as we weren’t planning to make any external connections to the EC2 server.
We did need to create a new security group, in order to be able to enable terminal access to the server.
For the same reason, we needed to assign a public IP address to the server, as that is needed for web console access to the server’s shell.
With this configured, we clicked Launch instance.
There was one more modification we needed to make to the server, so we clicked on the instance name at the top of the page to access the server instance details.
We needed to modify the firewall rules to allow access from the EC2 instance to our MSK cluster, so we needed to modify the security groups.
In the top-right menu, we navigated to Change security groups.
We chose the security group that was created for the MSK cluster.
Having both security groups gave our EC2 instance the permissions needed to let us connect from a web console, as well the permissions needed for it to connect to Kafka.
Next, we clicked Save.
We were now ready to access the instance, so we clicked Connect.
We went with EC2 Instance Connect as it offers a web-based terminal console with nothing to install.
To open the console, we clicked Connect.
This gave us a shell in our Amazon Linux server.
We started by installing Java, as this is required for running the Kafka admin tools.
sudo yum -y install java-11
Next, we downloaded Kafka.
curl -o kafka.tgz https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz tar -zxf kafka.tgz cd kafka_2.13-3.5.1
We matched the version of the MSK cluster (but it would have still worked if we had picked a different version).
To run the Kafka admin tools, we first needed to get connection details for the MSK cluster.
From the MSK cluster page, we clicked View client information.
We copied both the bootstrap address (labelled as the Private endpoint in the Bootstrap servers section) and the Plaintext ZooKeeper address.
Notice that as we hadn’t yet enabled public access, these were both private DNS addresses, but our EC2 server (running in the same VPC) would be able to access them.
We created BOOTSTRAP and ZOOKEEPER environment variables using our copied values.
export BOOTSTRAP=b-3.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9092,b-1.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9092,b-2.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9092 export ZOOKEEPER=z-1.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:2181,z-2.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:2181,z-3.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:2181
This was enough to let us create the topics we needed for our demonstration. We created a set of topics that are needed for the initial set of IBM Event Automation tutorials.
for TOPIC in ORDERS.NEW CANCELLATIONS DOOR.BADGEIN STOCK.MOVEMENT CUSTOMERS.NEW SENSOR.READINGS
do
./bin/kafka-topics.sh --create \
--bootstrap-server $BOOTSTRAP \
--replication-factor 3 \
--partitions 3 \
--config retention.bytes=25000000 \
--topic $TOPIC
done
You can see a list of these topics, together with a description of the events that we would be producing to them, in the documentation for the data generator we use for these demos.
With the topics created, it was time to start setting up authentication.
We started by setting up permissions for a user able to produce and consume events to all of our topics.
We named this user producer.
for TOPIC in ORDERS.NEW CANCELLATIONS DOOR.BADGEIN STOCK.MOVEMENT CUSTOMERS.NEW SENSOR.READINGS
do
./bin/kafka-acls.sh --add \
--authorizer-properties zookeeper.connect=$ZOOKEEPER \
--allow-principal "User:producer" \
--operation Write \
--topic $TOPIC
./bin/kafka-acls.sh --add \
--authorizer-properties zookeeper.connect=$ZOOKEEPER \
--allow-principal "User:producer" \
--operation Read \
--group="*" \
--topic $TOPIC
./bin/kafka-acls.sh --add \
--authorizer-properties zookeeper.connect=$ZOOKEEPER \
--allow-principal "User:producer" \
--operation Describe \
--group="*" \
--topic $TOPIC
done
Next, we set up more limited permissions, for a second user that would only be allowed to consume events from our topics.
We named this user consumer.
for TOPIC in ORDERS.NEW CANCELLATIONS DOOR.BADGEIN STOCK.MOVEMENT CUSTOMERS.NEW SENSOR.READINGS
do
./bin/kafka-acls.sh --add \
--authorizer-properties zookeeper.connect=$ZOOKEEPER \
--allow-principal "User:consumer" \
--operation Read \
--group="*" \
--topic $TOPIC
./bin/kafka-acls.sh --add \
--authorizer-properties zookeeper.connect=$ZOOKEEPER \
--allow-principal "User:consumer" \
--operation Describe \
--group="*" \
--topic $TOPIC
done
We didn’t have any Kafka administration left to do, so we were finished with this EC2 admin instance. In the interests of cleaning up, and keeping the demo running costs down, we terminated the EC2 instance and deleted the new security group we created for it.
Then we enabled public access to our MSK cluster.
▶
We started by turning on security. We went back to the MSK cluster instance page, and clicked the Properties tab.
We scrolled to the Security settings section and clicked Edit.
We enabled SASL/SCRAM authentication. This automatically enabled client TLS encryption as well.
We clicked Save changes.
We needed to wait for the cluster to update.
(This took over thirty minutes, so this was a good point to take a break!)
The prompt at the top of the MSK instance page showed that the next step was to set up secrets with username/passwords for Kafka clients to use.
We clicked Associate secrets.
We wanted two secrets: one for our producer user, the other for consumer user – each containing the username and password.
We clicked Create secret to get started.
We chose the Other type of secret, and used the Plaintext tab to create a JSON payload with a random password we generated for the producer user.
{
"username": "producer",
"password": "BE9rEMxwfC0eD7IQcVzC4s9csceBsKi3Enzi2wiY9B8uw73KsoNyR33vfFBKFozv"
}
We clicked Add new key to create a custom encryption key for this demo.
The default key type and usage were fine for our needs.
We gave it a name, again keeping with our “Loosehanger” clothing retailer scenario.
We set the permissions to make me the administrator of the key.
With the new key created, we could choose it to use this new encryption key for the secret with our new producer credentials.
To use the secret for Amazon MSK credentials, we needed to give the secret a name starting with AmazonMSK_.
We went with AmazonMSK_producer.
We clicked through the remaining steps until we could click Store.
We needed to repeat this to create a secret for our consumer user.
Again, we generated a long random password to use for clients that can only consume from our topics.
{
"username": "consumer",
"password": "RUkSRjUF6Nlw9420CCW50s3tdRTf3jq8R6Z0HQbneeUs8MiXtQ447OC003R538Nr"
}
We used the same new encryption key that we created for the consumer secret.
We had to give it a name with the same AmazonMSK_ prefix, so we went with the name AmazonMSK_consumer.
With these two secrets, our credentials were now ready.
We associated both of them with our MSK cluster.
We clicked Associate secrets.
With authentication prepared, we were ready to modify the MSK configuration to allow public access to the Kafka cluster.
First, we needed to modify the cluster configuration to set a property that Amazon requires for public access. We went to the Amazon MSK Cluster Configurations, clicked on our config name, then clicked Create revision.
We modified the following value:
allow.everyone.if.no.acl.found=false
And then clicked Create revision.
To use this modified configuration, we went back to the MSK cluster, and clicked on it.
We selected Edit cluster configuration.
We could then choose the new config revision, and click Save changes.
This can take ten minutes or so to complete.
Once this was complete, we clicked the Properties tab.
We scrolled to the Networking settings section, and selected the Edit public access option.
We enabled Turn on and clicked Save changes.
There was a bit of a wait for this change to be applied.
After thirty minutes, the configuration change was complete.
We now had an Amazon MSK cluster, with the topics we wanted to use, configured to allow public access, and with two SASL/SCRAM usernames/passwords prepared for our applications to use.
We started an app producing events to the Amazon MSK topics we’d created.
▶
First, we needed to modify our MSK cluster to allow connections from an app we would run on our laptop.
We went back to our MSK cluster instance, clicked on to the Properties tab, and scrolled to the Networking settings section.
We then clicked on the Security groups applied value.
On the security groups instance page, we clicked Edit inbound rules.
We needed to add a new rule for access to the Kafka port, 9196.
We could have added the specific source IP address for where we would run our app, but it was simpler for this quick demo to just allow access from applications running anywhere.
Our MSK cluster was now ready to allow connections.
To produce messages to our topics, we used a Kafka Connect connector. You can find the source connector we used at github.com/IBM/kafka-connect-loosehangerjeans-source. It is a data generator that periodically produces randomly generated messages, that we often use for giving demos.
To run Kafka Connect, we created a properties file called connect.properties.
We populated this with the following config. Note that the plugin.path location is a folder where we downloaded the source connector jar to – you can find the jar in the Releases page for the data gen source connector.
bootstrap.servers=b-1-public.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9196 security.protocol=SASL_SSL producer.security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 producer.sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="producer" password="BE9rEMxwfC0eD7IQcVzC4s9csceBsKi3Enzi2wiY9B8uw73KsoNyR33vfFBKFozv"; producer.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="producer" password="BE9rEMxwfC0eD7IQcVzC4s9csceBsKi3Enzi2wiY9B8uw73KsoNyR33vfFBKFozv"; client.id=loosehanger group.id=connect-group key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=false value.converter.schemas.enable=false offset.storage.file.filename=/tmp/connect/offsets plugin.path=/Users/dalelane/dev/demos/aws/connect/jars
We then created a properties file called connector.properties with the following config.
(You can see the other options we could have set in the connector README).
name=msk-loosehanger connector.class=com.ibm.eventautomation.demos.loosehangerjeans.DatagenSourceConnector
We ran it using connect-standalone.sh. (This is a script included in the bin folder of the Apache Kafka zip you can download from kafka.apache.org).
connect-standalone.sh connect.properties connector.properties
We left this running to create a live stream of events that we could use for demos.
We now had streams of events on MSK topics, ready to process using IBM Event Processing.
▶
We started by following the Transform events to create or remove properties tutorial from ibm.github.io/event-automation/tutorials.
We were running an instance of IBM Event Processing in an OpenShift cluster in IBM Cloud. (For details of how we deployed this, you can see the first step in the tutorial instructions. It just involved running an ansible playbook).
We started by logging on to the Event Processing authoring dashboard.
And we started to create a new flow.
We created an event source in Event Processing using an Amazon MSK topic.
▶
We went to the MSK instance page, and clicked View client information. From there, we copied the public bootstrap address.
We pasted that into the Server box in the event source configuration page. We needed to split up the comma-separated address we got from the Amazon MSK page, as Event Processing requires separate broker addresses.
We provided the consumer credentials we created earlier when setting up the MSK cluster.
The list of topics displayed in Event Processing matches the list of topics that we configured the consumer user to have access to.
Following the tutorial instructions, we chose the ORDERS.NEW topic.
We copied in a sample message from the ORDERS.NEW topic.
Finally, we created an event processing flow using the events from an MSK topic.
▶
We continued to create the flow as described in the tutorial instructions.
All of the Event Processing tutorials can be followed as written using the Amazon MSK cluster that we created.
Next, we demonstrated the value that Event Endpoint Management can bring to an Amazon MSK cluster. We showed how adding MSK topics to a self-service catalog enables sharing and reuse of existing topics, wherever they are hosted. And we showed the way that the addition of an Event Gateway can maintain control and governance of these topics as they are shared.
The diagram above is a simplified description of what we created. We used the same MSK cluster in AWS that we had used for the previous demo, as it already had a variety of topics and a data generator producing live streams of events to them. This time we used it with an instance of Event Endpoint Management (that was running in our Red Hat OpenShift cluster in IBM Cloud).
We added our Amazon MSK topics to the catalog, and configured an Event Gateway to govern access to them. We could have run the Event Gateway in OpenShift, alongside the endpoint manager. However, for this demonstration, we wanted to show the flexibility of running the Event Gateway in the same environment as a Kafka cluster. This showed how you can remove the need for egress from the AWS environment where your Kafka applications are also running in AWS.
Finally, we showed all of this in action by running a Kafka consumer, consuming events from the MSK topics. The consumer was using credentials created in the Event Endpoint Management catalog and connected via the Event Gateway.
How we created a demonstration of this…
We started by creating a security group (for the load balancer that would provide the external connection for the Event Gateway).
▶
We started by going to the Security Groups section of the EC2 service in AWS.
We clicked Create security group.
We created a security group that would accept connections on port 443.
Next, we went to the IP Target groups page and clicked Create target group.
We chose IP addresses as the target type, and gave it a name that explained this would be the target address for the Event Gateway.
We chose port 443. Other port numbers would have been fine, but this is consistent with the port number used to access the Event Gateway when it is running in OpenShift.
For the VPC, we chose the same VPC that the MSK cluster is in.
Next, we defined a healthcheck. Again, we did this to be consistent with the way the Event Gateway runs when managed by the Operator in OpenShift, by using the same protocol (HTTP), port number (8081), and path (/ready) that are used for probes in Kubernetes.
Then we clicked Next.
The default values are fine for the next Register targets step, so we clicked Create target group.
Then, we created the network load balancer to give the Event Gateway an external address.
▶
We started by going to the Load balancers page within EC2, and clicked Create load balancer.
We chose Network Load Balancer and clicked Create.
We gave it a name, and made it Internet-facing because we wanted the event gateway to be accessible from outside of AWS.
We selected one of the three availability zones created when we set up the MSK cluster to put the gateway in.
Then we selected security groups for the load balancer to use: the security group created for the MSK cluster, and the new one created for the gateway.
That was everything we needed, so we clicked Create load balancer.
It took a little while to complete the setup.
We waited for it to no longer be in a Provisioning state before continuing.
Now that we had an external DNS address, we could create a certificate for the Event Gateway.
▶
We clicked into the network load balancer that we’d just created to view the DNS name that we would give to the gateway.
We’d installed IBM Event Endpoint Management using the demo ansible playbook. This setup uses a Kubernetes Certificate Manager in OpenShift to setup the SSL/TLS certs needed. This meant the issuer and CA we needed to create a certificate for the Event Gateway we would run in AWS was in OpenShift.
Our next step was to go into OpenShift so that we could create a new cert.
We just needed to copy the DNS name for the gateway to use in the cert.
That gave us what we needed to create the certificate.
Notice that we used the DNS name in two places: spec.dnsNames and spec.uris. You can see more information about the requirements for this certificate in the Event Endpoint Management documentation.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: amazon-eem-gw-cert
namespace: event-automation
spec:
dnsNames:
- loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com
duration: 2160h0m0s
issuerRef:
kind: Issuer
name: my-eem-gateway-ibm-egw-iss
privateKey:
algorithm: RSA
rotationPolicy: Always
secretName: amazon-eem-gw-cert
subject:
organizations:
- IBM Event Endpoint Management
uris:
- egw://loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com:443/amazon-gateways/eu-west-1a
usages:
- client auth
- digital signature
- server auth
The cert manager created the certificate based on this spec, and stored it in a Kubernetes secret in the event-automation namespace where Event Endpoint Management was running.
We downloaded ca.crt, tls.crt, and tls.key to three files that would be needed to run the gateway.
To make the certificates available to the Event Gateway we would run in AWS, we created a container image that would hold the certificates.
▶
We decided to use a custom container image to hold the certificates that we’d just created. We wanted to store this as a private image in Amazon’s container registry, so we went to the ECR service in AWS.
Then we clicked Get started.
We chose the repository name event-gateway-certs and set the visibility to Private.
That gave us an image name that we could push a container image to. We clicked on the View push commands button to get the details for this.
First we needed to copy the login command.
We used this to build and push a custom container image to ECR. Our Dockerfile looked like this:
FROM registry.access.redhat.com/ubi8/ubi-minimal:latest COPY ca.crt /certs/eem/ca.pem COPY tls.crt /certs/eem/client.pem COPY tls.key /certs/eem/client.key COPY tls.crt /certs/eem/egwclient.pem COPY tls.key /certs/eem/egwclient-key.pem VOLUME ["/certs"]
The role of this container image would simply be to make the certificate files that we had downloaded in the previous step available to the Event Gateway.
More information about the certificate file names we used, and the way these files would be used, can be found in the documentation for running stand-alone Event Gateways.
We then created another security group, this time for the Event Gateway.
▶
The previous security group we had created was for the network load balancer that would provide the external address for the Event Gateway. That was a security group that needed to allow connections from external clients.
Now we needed to create a security group for the Event Gateway container itself, which would need to receive connections from the load balancer, and make connections to the Amazon MSK cluster.
We started by going back to the Security Groups page.
We gave the security group a name that explained what this would be used for.
We defined two inbound rules.
The first was for the healthcheck address we had defined previously. This does not return sensitive data, so we allowed connections to this from anywhere, which would allow both the load balancer and ECS to use the healthcheck.
The second was for the port used for Kafka traffic coming from the network load balancer, so we specified the 8092 port number and the security group for the network load balancer.
The security group was now ready to use.
We created a task definition for running the Event Gateway in ECS.
▶
We went to the Elastic Container Service next, to start creating a new task definition for the Event Gateway.
We gave our task definition the name loosehanger-event-gateway.
We opted for a launch type of AWS Fargate as that was a serverless option that would let us run the Event Gateway when it was being used.
We chose Linux/X86/64 and a relatively small task size for our quick demo.
We also needed to create a Task execution role to run the container. It needed to be able to pull the Event Gateway from the IBM Entitled Registry, so the task execution role needed to have access to credentials for this.
To enable this, we opened the AWS Secrets Manager so that we could Store a new secret to hold an Entitled Registry pull secret.
We created a new entitlement key from myibm.ibm.com/products-services/containerlibrary.
We used this to create a new Other type of secret using the Plaintext tab to create a secret like this:
{
"username": "cp",
"password": "our-entitled-registry-key"
}
Note that your entitled registry key is a password that you shouldn’t share publicly. The key you can see in the screenshot above has since been revoked.
We encrypted this secret using the encryption key that we’d created earlier.
We gave the secret a helpful description to remind us what this secret contains.
The secret was ready to use. To use it in our task definition, we clicked into the instance to get the ARN for it.
We copied the ARN for the secret – so that we could add this to the container definition as the pull secret.
Back in the ECS task definition, we started filling in the details of the Event Gateway container.
As described in the Event Endpoint Management documentation for installing a stand-alone gateway, we needed to use an Image URI of cp.icr.io/cp/ibm-eventendpointmanagement/egw:11.0.5
We marked this as an Essential container, and added two port mappings: one for the healthcheck, and one for Kafka traffic.
We pasted the pull secret ARN in for pulling the Event Gateway image.
The ID of this isn’t sufficient for it to be used – the task execution role also needed permission to access the pull secret and decrypt it.
To do this, once the task execution role was created, we needed to attach some additional policies. Firstly, we needed to give it permission to pull our custom private certificates image from ECR.
In order for the task execution role to decrypt the Entitled Registry pull secret, we needed to give it details of the loosehanger-keys encryption key we’d created earlier. We started by getting the ARN for this key.
In order to pull the Event Gateway image from the Entitled Registry, the task execution role needs to be access:
* the Entitled Registry we’d stored in Secrets Manager
* the decryption key needed to be able to decrypt that secret
We added the ARNs for both of these to a custom permissions policy.
Finally, we needed to set some environment variables to add to the Event Gateway container. To get the right values for this, we returned to the Event Endpoint Management manager in OpenShift.
We needed to copy the API endpoint URI.
This provided the value for the backendURL environment variable, which we set to https://my-eem-manager-ibm-eem-gateway-event-automation.itzroks-120000f8p4-ivahj5-6ccd7f378ae819553d37d5f2ee142bd6-0000.eu-gb.containers.appdomain.cloud
We set the value of the GATEWAY_PORT environment variable to 8092 as that is the port number we had chosen for Kafka traffic in the port mappings.
We set the value of the GATEWAY_HEALTH_PORT environment variable to 8081 as that is the port number we had chosen for the healthcheck in the port mappings.
We set the value of the KAFKA_ADVERTISED_LISTENER environment variable to loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com:443 as it was the DNS name we had created for the Event Gateway load balancer.
We set the value of the certPaths environment variable to this, as it matched the locations we’d created in the custom certs container we had built.
/certs/eem/client.pem,/certs/eem/client.key,/certs/eem/ca.pem,/certs/eem/egwclient.pem,/certs/eem/egwclient-key.pem
Finally, we added a few additional environment variables to match the way the gateway is run when managed by the Operator in OpenShift.
GATEWAY_REGISTRATION_SPEC_LOCATION /opt/ibm/gateway/openapi-specs/gw-director-openapi.yaml externalGatewayConfigJsonFile /config/gatewayConfig.json features eemgw,core
These were all of the environment variables that we needed.
We enabled logging, as it can be useful when identifying what the gateway is doing.
We defined a healthcheck action (a curl of the gateway’s readiness URL) for the container. Again, we were following the pattern for the probes that are set up when the Operator runs the Event Gateway in Kubernetes.
Finally, we needed to add the certificates we created for the gateway available by adding the custom certificates container we built as a secondary container.
We gave the certs container a command so that ECS wouldn’t treat it as a crashed container. We didn’t need this container to run anything, we just needed it to share a volume with the Event Gateway.
We achieved this by giving the gateway container read-only access to the storage from the certificates container.
This was everything we needed to complete the task specification, so at this point we clicked Create.
Now we were ready to create an ECS cluster to run our Event Gateway task.
▶
We were ready to run the Event Gateway task, so we went to the Clusters page in ECS and clicked Create cluster.
We defined a Fargate cluster and gave it a name.
Then we waited it for it to be provisioned.
Once it was ready, we clicked into the cluster instance, went to the Services tab, and clicked the Create button.
For the compute options, we opted for the Launch type compute option – as we only wanted to run a single instance of the gateway.
We chose our Event Gateway task definition, and gave the service a name.
We chose the same availability zone to run the Event Gateway in that we’d chosen when creating the network load balancer.
Then we chose the security group we had created for the gateway container.
We chose the load balancer we had created for the gateway.
For the listener, we used the listener we had defined when creating the load balancer.
Finally, we clicked Create.
Now we just needed to wait for the Event Gateway to start.
We verified that the Event Gateway running in AWS had connected to the Endpoint manager running in OpenShift.
▶
A good way to know when the Event Gateway is ready is to check the Gateways tab in the Event Endpoint Management manager. We could see our new Event Gateway listed in the table (alongside the existing Event Gateway we had created in OpenShift in IBM Cloud).
We could now start adding Amazon MSK topics to the Event Endpoint Management catalog.
▶
First, we went to the topics tab in the Event Endpoint Management manager.
We needed to add the Amazon MSK cluster first, so we clicked Add new cluster.
We gave this new Kafka cluster a name: Amazon MSK.
We needed to get the bootstrap address for the MSK cluster, using the same View client information button as before. Note that we needed to split the public bootstrap address we got from Amazon into separate broker addresses.
We needed to accept the certificates when prompted.
We used the consumer credentials that we had created for the MSK cluster, which had permission to consume from all of our application topics.
With the credentials accepted, we were now ready to start adding topics.
We confirmed that all of the topics our consumer credentials had access to were listed.
We selected all of them. We could have given them different aliases, but we decided to use the existing topic names as-is.
Our topics were now added to the catalog, ready for additional documentation and publishing.
For each topic, we needed to write additional documentation.
We added a recent message from the topic as a sample, to make the catalog documentation more useful.
Finally, we could click Publish.
We needed to choose the gateway groups we wanted to be able to consume this topic through. We chose only the Event Gateway running in AWS.
Our topic was published! We were now able to repeat this for our other Amazon MSK topics.
Finally, to show this all working, we created a Kafka consumer to receive events from Amazon MSK topics through the Event Gateway.
▶
From the Event Endpoint Management catalog, we could review what people finding our topics in the catalog would see.
To show this in action, we created credentials for consuming from the topic.
This gave us a unique username and password for consuming from this topic through the Event Gateway.
We clicked Download certificates to download the Event Gateway certificate as a PEM file.
We copied the bootstrap address from the Servers section of the Catalog page.
We put all of this: the location of the downloaded certificate file, the bootstrap address, and username and password we had generated from the Catalog, into a properties file. It looked like this:
bootstrap.servers=loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com:443 security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="eem-582bdccb-f1b4-479d-940a-7480147e70b1" password="018c80b7-218b-4eed-a8f5-169d2cb64206"; ssl.truststore.location=/Users/dalelane/Downloads/certificate_loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com_443.pem ssl.truststore.type=PEM group.id=test-consumer-group enable.auto.commit=false
We could use this from kafka-console-consumer.sh to start consuming events through the Event Gateway.
Finally, we demonstrated how these could be combined, bringing the value of both the previous demonstrations together.
Making Amazon MSK topics available through a self-service catalog can enable much wider reuse of these streams of events. And providing a low-code authoring canvas for processing these events can extend this use beyond just developers, enabling both business and IT teams to define the scenarios they need to respond to.
For this final demonstration, we again used the same Amazon MSK cluster, with the same range of topics and live streams of events as before. We had already added these to the Event Endpoint Management catalog for the previous demo, so for this demonstration we showed how MSK topics found in the catalog can easily be used in Event Processing to quickly identify new real-time insights.
How we created a demonstration of this…
We demonstrated an Event Endpoint Management catalog can contain topics from many distributions of Kafka cluster, including topics from Amazon MSK.
▶
We had published two of the topics so far to be able to demonstrate Event Endpoint Management.
With them both available in the catalog, that was enough to demonstrate another of the Event Processing tutorials.
First, we started by showing that we could still add topics to the catalog from non-MSK Kafka clusters. We clicked Add topic and then chose an Event Streams cluster.
This was useful to show the benefits of aliases. If you have topics with the same name in different Kafka clusters, and want to add both to the catalog, aliases let you differentiate them.
As the owner of the topic, we could see which topic(s) were hosted on which Kafka clusters.
However, for someone finding the topics in the catalog, they all just look like Kafka topics. You can use tags to identify where the topics are from if you want developers to know, as we had done here.
Next, to process these topics using IBM Event Processing, we created a new flow.
▶
We went back to the Event Processing UI, running in our OpenShift cluster in IBM Cloud.
We created a new event processing flow and gave it a name.
We defined event sources based on Amazon MSK topics, using details from the Event Endpoint Management catalog.
▶
We created an event source, using the bootstrap address we copied from the Event Endpoint Management catalog page.
We needed to accept the certificates as before.
We used credentials that we created in the Event Endpoint Management catalog.
Then we confirmed the topic name.
We provided the sample message from the catalog to give Event Processing information about the properties of messages on this topic.
Then we started repeating this to add another event source.
This second event source was also based on an Amazon MSK topic, that we discovered in the Event Endpoint Management catalog and accessed via the Event Gateway running in AWS.
Again, we provided the bootstrap address for the Event Gateway that we copied from the catalog page.
We needed to confirm the certificates.
We used the Event Endpoint Management catalog to generate credentials unique for this second topic.
We added these credentials to the Event Processing configuration.
Then we confirmed the topic name.
And finally provided the sample message to give Event Processing info about the properties of messages on this second topic.
Finally, we added an interval join to correlate events from these two MSK topics.
▶
We used an Interval join to join these two MSK topics.
We gave this join a name.
We used the assistant to help us identify the attributes of each stream of events that we could join on.
We used the visualisation to help us specify the join window.
Then we chose, and named, the attributes from each stream of events that we wanted to output.
Finally, to show everything working, we ran the flow and viewed the results.
Our goal with this blog post was to demonstrate what can be done with IBM Event Automation, with a particular focus on the benefits of composability. By taking advantage of the de-facto standard nature of the Kafka protocol, we can layer additional capabilities on top of Apache Kafka clusters, wherever they are running.
Our demonstrations were intended to provide an illustrative example of using Event Automation with MSK. It was absolutely not meant to be a description of how to use Amazon services in a perfect or optimum way, but instead focused on a quick and simple way to show what is possible. We wanted to inspire you for how you could get more out of your own Amazon MSK cluster.
For more information about any of the ideas that we have shared here, please see the Event Automation documentation, or get in touch.
Amazon Web Services (AWS), Amazon Managed Streaming for Kafka (MSK), Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Container Registry (ECR) are trademarks of Amazon Web Services. This blog post is intended to explain how we demonstrated the use of these services, but does not imply any affiliation with, or endorsement by, Amazon or AWS.
Tags: apachekafka, eventautomation, kafka