{"id":5023,"date":"2023-10-25T16:57:15","date_gmt":"2023-10-25T16:57:15","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5023"},"modified":"2023-10-25T16:57:16","modified_gmt":"2023-10-25T16:57:16","slug":"using-ibm-event-automation-with-amazon-msk","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5023","title":{"rendered":"Using IBM Event Automation with Amazon MSK"},"content":{"rendered":"

Written with Chris Patmore<\/a><\/em><\/p>\n

IBM Event Automation<\/a> helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have.<\/p>\n

A key goal of Event Automation is to be composable. The three components can be used together, or they can each be used to extend and enhance an existing event-driven deployment.<\/p>\n

Amazon MSK (Managed Streaming for Kafka) is a hosted, managed Kafka service available in Amazon Web Services. If a business has started their event-driven journey using MSK, then components from Event Automation can help to enhance this. This could be by offering management and governance of their MSK topics. And it could be by providing an intuitive low-code authoring canvas to process the events on their MSK topics.<\/p>\n

Working with Amazon MSK is a nice example of the benefits of the composability of Event Automation, by helping businesses to get more value from their existing MSK topics.<\/p>\n

In this blog post, we want to show a few different examples of where this can be done. For each example, we’ll provide a high-level diagram and description. We’ll also share a demonstration that we created to show it in action.<\/p>\n

(Click on the descriptions for detailed step-by-step instructions and screenshots of how we built each demo.)<\/em><\/p>\n

\n

\"using<\/a><\/p>\n

\n To start with, we demonstrated how Event Processing<\/strong> can be used with Amazon MSK. We showed how Event Processing, based on Apache Flink, can help businesses to identify insights from the events on their MSK topics through an easy-to-use authoring canvas.<\/p>\n

The diagram above is a simplified description of what we created. We created an MSK cluster in AWS, set up a few topics, and then started a demonstration app producing a stream of events to them. This gave us a simplified example of a live MSK cluster.<\/p>\n

We then accessed this Amazon MSK cluster from an instance of Event Processing (that was running in a Red Hat OpenShift cluster in IBM Cloud). We used Event Processing to create a range of stateful stream processing flows.<\/p>\n

This showed how the events on MSK topics can be processed where they are, without requiring an instance of Event Streams or for topics to be mirrored into a Kafka cluster also running in OpenShift. Using the low-code authoring canvas with Kafka topics that you already have, wherever they are, is a fantastic event-driven architecture enabler.<\/p>\n

\n\n How we created a demonstration of this…
\n<\/summary>\n
\n\n We started by creating an Amazon MSK cluster.
\n▶<\/span>
\n<\/summary>\n
\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

We opened the Amazon Web Services admin console, and went to the MSK (Managed Streaming for Apache Kafka) service.<\/p>\n

To start, we clicked Create cluster<\/strong>.<\/p>\n

\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

We opted for the Custom create<\/strong> option so we could customize our MSK cluster.<\/p>\n

We called the cluster loosehanger-msk<\/code> because we’re basing this demonstration on “Loosehanger” – a fictional clothes retailer that we have a data generator<\/a> for.<\/p>\n

\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

We chose a Provisioned<\/strong> (rather than serverless) Kafka cluster type, and chose the latest version of Kafka that Amazon offered (3.5.1<\/strong>).<\/p>\n

Because we only needed an MSK cluster for this short demo, we went with the small<\/strong> broker type.<\/p>\n

\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

We went with the default, and recommended, number of zones to distribute the Kafka brokers across: three<\/strong>.<\/p>\n

Because we only planned to run a few applications with a small number of topics, we didn’t need a lot of storage – we gave each broker 5GB<\/strong> of storage.<\/p>\n

\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

Rather than go with the default Kafka configuration, we clicked Create configuration<\/strong> to prepare a new custom config.<\/p>\n

\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

We gave the config a name similar to the MSK cluster itself, based on our scenario of the fictional clothes retailer<\/a>, Loosehanger.<\/p>\n

\n
\n <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n

We started with a config that would make it easy for us to set up the topics.<\/p>\n

auto.create.topics.enable=true\ndefault.replication.factor=3\nmin.insync.replicas=2\nnum.io.threads=8\nnum.network.threads=5\nnum.partitions=1\nnum.replica.fetchers=2\nreplica.lag.time.max.ms=30000\nsocket.receive.buffer.bytes=102400\nsocket.request.max.bytes=104857600\nsocket.send.buffer.bytes=102400\nunclean.leader.election.enable=true\nzookeeper.session.timeout.ms=18000\nallow.everyone.if.no.acl.found=true<\/pre>\n
The key value we added to the default config was the allow.everyone.if.no.acl.found<\/code> one, to make it clear that we would start creating topics before setting up auth or access control lists.<\/p>\n
We clicked Create<\/strong> to create this configuration. Once back on the MSK cluster settings screen, we chose this custom config and clicked Next<\/strong> to move on to the next step.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Networking was next.<\/p>\n
We clicked Create VPC<\/strong> to prepare a new virtual networking environment for our demonstration.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We opted for the VPC and more<\/strong> option so we could set this up in a way that would support public access to the MSK cluster.<\/p>\n
We chose three availability zones<\/strong> to match the config we used for the MSK cluster – this would allow us to have a separate AZ for each MSK broker.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We went with three public subnets and no private subnets, again as this would allow us to enable public access to the MSK cluster.<\/p>\n
We left the default DNS options enabled so that we could have DNS hostnames for our addresses.<\/p>\n
Next we clicked Create VPC<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We verified that the VPC resources we requested were created and then closed this window.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Back on the Networking<\/strong> step of the MSK cluster creation wizard, we were now able to choose our new VPC, and select the zones and subnets. The match of three availability zones for the MSK cluster, and three availability zones for the VPC meant it was just a matter of choosing a different zone for each broker.<\/p>\n
We wanted to enable public access, but this can’t be done at cluster creation time, so this remained on Off<\/strong> for now.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The final networking step option is to create security groups. The default option here was fine for our purposes, so we left this as-is.<\/p>\n
We clicked Next<\/strong> to move onto the next step.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The next step was to configure the security options for the MSK cluster.<\/p>\n
We started with disabling auth, as unauthenticated access would make it easy for us to set up our cluster and topics. We enabled auth after we had the cluster the way we wanted it.<\/p>\n
For the same reason, we also left client TLS disabled as well. We turned this on later when we enabled public access to the cluster.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The default encryption key was fine for encrypting the broker storage, so we left this as-is and clicked Next<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The next step is to configure monitoring. As a short-lived demo cluster, we didn’t have monitoring requirements, so we left this on the default basic option and clicked Next<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Our MSK cluster specification was ready to go, so we clicked Create<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We had to wait for the MSK cluster to provision.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Once it was ready, we could move on to the next stage which was to create topics.<\/p>\n<\/details>\n
\n\n                        Next, we created some Kafka topics that we would use with Event Automation, and set up some credentials for accessing them.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Amazon MSK doesn’t offer admin controls for the Kafka cluster, so we needed somewhere that we could run a Kafka admin client.<\/p>\n
The simplest option was to create an EC2 server where we could run Kafka admin commands from. We went to the EC2 service within AWS, and clicked Launch instance<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We kept with our naming theme of “Loosehanger” to represent our fictional clothes retailer.<\/p>\n
This would only be a short-lived server, that we would keep around long enough to run a few Kafka admin commands, so quick and simple was the priority. We went with the Amazon Linux<\/strong> quick start option.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Again, as this would only be a short-lived server running a small number of command-line scripts, a small, free-tier-eligible instance type was fit for our needs.<\/p>\n
We didn’t need to create a key pair as we weren’t planning to make any external connections to the EC2 server.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We did need to create a new security group, in order to be able to enable terminal access to the server.<\/p>\n
For the same reason, we needed to assign a public IP address to the server, as that is needed for web console access to the server’s shell.<\/p>\n
With this configured, we clicked Launch instance<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
There was one more modification we needed to make to the server, so we clicked on the instance name at the top of the page to access the server instance details.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to modify the firewall rules to allow access from the EC2 instance to our MSK cluster, so we needed to modify the security groups.<\/p>\n
In the top-right menu, we navigated to Change security groups<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose the security group that was created for the MSK cluster.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Having both security groups gave our EC2 instance the permissions needed to let us connect from a web console, as well the permissions needed for it to connect to Kafka.<\/p>\n
Next, we clicked Save<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We were now ready to access the instance, so we clicked Connect<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We went with EC2 Instance Connect<\/strong> as it offers a web-based terminal console with nothing to install.<\/p>\n
To open the console, we clicked Connect<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This gave us a shell in our Amazon Linux server.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We started by installing Java, as this is required for running the Kafka admin tools.<\/p>\n
sudo yum -y install java-11<\/pre>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Next, we downloaded Kafka.<\/p>\n
curl -o kafka.tgz https:\/\/downloads.apache.org\/kafka\/3.5.1\/kafka_2.13-3.5.1.tgz\ntar -zxf kafka.tgz\ncd kafka_2.13-3.5.1<\/pre>\n
We matched the version of the MSK cluster (but it would have still worked if we had picked a different version).<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To run the Kafka admin tools, we first needed to get connection details for the MSK cluster.<\/p>\n
From the MSK cluster page, we clicked View client information<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We copied both the bootstrap address (labelled as the Private endpoint<\/strong> in the Bootstrap servers<\/strong> section) and the Plaintext ZooKeeper<\/strong> address.<\/p>\n
Notice that as we hadn’t yet enabled public access, these were both private DNS addresses, but our EC2 server (running in the same VPC) would be able to access them.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We created BOOTSTRAP<\/code> and ZOOKEEPER<\/code> environment variables using our copied values.<\/p>\n
export BOOTSTRAP=b-3.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9092,b-1.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9092,b-2.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9092\nexport ZOOKEEPER=z-1.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:2181,z-2.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:2181,z-3.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:2181<\/pre>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This was enough to let us create the topics we needed for our demonstration. We created a set of topics that are needed for the initial set of IBM Event Automation tutorials<\/a>.<\/p>\n
for TOPIC in ORDERS.NEW CANCELLATIONS DOOR.BADGEIN STOCK.MOVEMENT CUSTOMERS.NEW SENSOR.READINGS\ndo\n    .\/bin\/kafka-topics.sh --create \\\n        --bootstrap-server $BOOTSTRAP \\\n        --replication-factor 3 \\\n        --partitions 3 \\\n        --config retention.bytes=25000000 \\\n        --topic $TOPIC\ndone<\/pre>\n
You can see a list of these topics, together with a description of the events that we would be producing to them, in the documentation for the data generator<\/a> we use for these demos.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
With the topics created, it was time to start setting up authentication.<\/p>\n
We started by setting up permissions for a user able to produce and consume events to all of our topics.<\/p>\n
We named this user producer<\/code>.<\/p>\n
for TOPIC in ORDERS.NEW CANCELLATIONS DOOR.BADGEIN STOCK.MOVEMENT CUSTOMERS.NEW SENSOR.READINGS\ndo\n    .\/bin\/kafka-acls.sh --add \\\n        --authorizer-properties zookeeper.connect=$ZOOKEEPER \\\n        --allow-principal \"User:producer\" \\\n        --operation Write \\\n        --topic $TOPIC\n    .\/bin\/kafka-acls.sh --add \\\n        --authorizer-properties zookeeper.connect=$ZOOKEEPER \\\n        --allow-principal \"User:producer\" \\\n        --operation Read \\\n        --group=\"*\" \\\n        --topic $TOPIC\n    .\/bin\/kafka-acls.sh --add \\\n        --authorizer-properties zookeeper.connect=$ZOOKEEPER \\\n        --allow-principal \"User:producer\" \\\n        --operation Describe \\\n        --group=\"*\" \\\n        --topic $TOPIC\ndone<\/pre>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Next, we set up more limited permissions, for a second user that would only be allowed to consume events from our topics.<\/p>\n
We named this user consumer<\/code>.<\/p>\n
for TOPIC in ORDERS.NEW CANCELLATIONS DOOR.BADGEIN STOCK.MOVEMENT CUSTOMERS.NEW SENSOR.READINGS\ndo\n    .\/bin\/kafka-acls.sh --add \\\n        --authorizer-properties zookeeper.connect=$ZOOKEEPER \\\n        --allow-principal \"User:consumer\" \\\n        --operation Read \\\n        --group=\"*\" \\\n        --topic $TOPIC\n    .\/bin\/kafka-acls.sh --add \\\n        --authorizer-properties zookeeper.connect=$ZOOKEEPER \\\n        --allow-principal \"User:consumer\" \\\n        --operation Describe \\\n        --group=\"*\" \\\n        --topic $TOPIC\ndone<\/pre>\n
We didn’t have any Kafka administration left to do, so we were finished with this EC2 admin instance. In the interests of cleaning up, and keeping the demo running costs down, we terminated the EC2 instance and deleted the new security group we created for it.<\/p>\n<\/details>\n
\n\n                        Then we enabled public access to our MSK cluster.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We started by turning on security. We went back to the MSK cluster instance page, and clicked the Properties<\/strong> tab.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We scrolled to the Security settings<\/strong> section and clicked Edit<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We enabled SASL\/SCRAM authentication<\/strong>. This automatically enabled client TLS encryption as well.<\/p>\n
We clicked Save changes<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to wait for the cluster to update.<\/p>\n
(This took over thirty minutes, so this was a good point to take a break!)<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The prompt at the top of the MSK instance page showed that the next step was to set up secrets with username\/passwords for Kafka clients to use.<\/p>\n
We clicked Associate secrets<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We wanted two secrets: one for our producer<\/code> user, the other for consumer<\/code> user – each containing the username and password.<\/p>\n
We clicked Create secret<\/strong> to get started.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose the Other<\/strong> type of secret, and used the Plaintext<\/strong> tab to create a JSON payload with a random password we generated for the producer user.<\/p>\n
{\n    \"username\": \"producer\",\n    \"password\": \"BE9rEMxwfC0eD7IQcVzC4s9csceBsKi3Enzi2wiY9B8uw73KsoNyR33vfFBKFozv\"\n}<\/pre>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We clicked Add new key<\/strong> to create a custom encryption key for this demo.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The default key type and usage were fine for our needs.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave it a name, again keeping with our “Loosehanger” clothing retailer scenario.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We set the permissions to make me the administrator of the key.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
With the new key created, we could choose it to use this new encryption key for the secret with our new producer credentials.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To use the secret for Amazon MSK credentials, we needed to give the secret a name starting with AmazonMSK_<\/code>.<\/p>\n
We went with AmazonMSK_producer<\/code>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We clicked through the remaining steps until we could click Store<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to repeat this to create a secret for our consumer user.<\/p>\n
Again, we generated a long random password to use for clients that can only consume from our topics.<\/p>\n
{\n    \"username\": \"consumer\",\n    \"password\": \"RUkSRjUF6Nlw9420CCW50s3tdRTf3jq8R6Z0HQbneeUs8MiXtQ447OC003R538Nr\"\n}<\/pre>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used the same new encryption key that we created for the consumer secret.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We had to give it a name with the same AmazonMSK_<\/code> prefix, so we went with the name AmazonMSK_consumer<\/code>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
With these two secrets, our credentials were now ready.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We associated both of them with our MSK cluster.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We clicked Associate secrets<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
With authentication prepared, we were ready to modify the MSK configuration to allow public access to the Kafka cluster.<\/p>\n
First, we needed to modify the cluster configuration to set a property that Amazon requires for public access. We went to the Amazon MSK Cluster Configurations, clicked on our config name, then clicked Create revision<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We modified the following value:<\/p>\n
allow.everyone.if.no.acl.found=false<\/pre>\n
And then clicked Create revision<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To use this modified configuration, we went back to the MSK cluster, and clicked on it.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We selected Edit cluster configuration<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We could then choose the new config revision, and click Save changes<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This can take ten minutes or so to complete.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Once this was complete, we clicked the Properties<\/strong> tab.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We scrolled to the Networking settings<\/strong> section, and selected the Edit public access<\/strong> option.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We enabled Turn on<\/strong> and clicked Save changes<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
There was a bit of a wait for this change to be applied.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
After thirty minutes, the configuration change was complete.<\/p>\n
We now had an Amazon MSK cluster, with the topics we wanted to use, configured to allow public access, and with two SASL\/SCRAM usernames\/passwords prepared for our applications to use.<\/p>\n<\/details>\n
\n\n                        We started an app producing events to the Amazon MSK topics we’d created.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
First, we needed to modify our MSK cluster to allow connections from an app we would run on our laptop.<\/p>\n
We went back to our MSK cluster instance, clicked on to the Properties<\/strong> tab, and scrolled to the Networking settings<\/strong> section.<\/p>\n
We then clicked on the Security groups applied<\/strong> value.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
On the security groups instance page, we clicked Edit inbound rules<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to add a new rule for access to the Kafka port, 9196<\/code>.<\/p>\n
We could have added the specific source IP address for where we would run our app, but it was simpler for this quick demo to just allow access from applications running anywhere.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Our MSK cluster was now ready to allow connections.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To produce messages to our topics, we used a Kafka Connect connector. You can find the source connector we used at github.com\/IBM\/kafka-connect-loosehangerjeans-source<\/a>. It is a data generator that periodically produces randomly generated messages, that we often use for giving demos.<\/p>\n
To run Kafka Connect, we created a properties file called connect.properties<\/code>.<\/p>\n
We populated this with the following config. Note that the plugin.path<\/code> location is a folder where we downloaded the source connector jar to – you can find the jar in the Releases page<\/a> for the data gen source connector.<\/p>\n
bootstrap.servers=b-1-public.loosehangermsk.krrnez.c3.kafka.eu-west-1.amazonaws.com:9196\n\nsecurity.protocol=SASL_SSL\nproducer.security.protocol=SASL_SSL\n\nsasl.mechanism=SCRAM-SHA-512\nproducer.sasl.mechanism=SCRAM-SHA-512\n\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"producer\" password=\"BE9rEMxwfC0eD7IQcVzC4s9csceBsKi3Enzi2wiY9B8uw73KsoNyR33vfFBKFozv\";\nproducer.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"producer\" password=\"BE9rEMxwfC0eD7IQcVzC4s9csceBsKi3Enzi2wiY9B8uw73KsoNyR33vfFBKFozv\";\n\nclient.id=loosehanger\ngroup.id=connect-group\n\nkey.converter=org.apache.kafka.connect.storage.StringConverter\nvalue.converter=org.apache.kafka.connect.json.JsonConverter\nkey.converter.schemas.enable=false\nvalue.converter.schemas.enable=false\noffset.storage.file.filename=\/tmp\/connect\/offsets\nplugin.path=\/Users\/dalelane\/dev\/demos\/aws\/connect\/jars<\/pre>\n
We then created a properties file called connector.properties<\/code> with the following config.<\/p>\n
(You can see the other options we could have set in the connector README<\/a>).<\/p>\n
name=msk-loosehanger\nconnector.class=com.ibm.eventautomation.demos.loosehangerjeans.DatagenSourceConnector<\/pre>\n
We ran it using connect-standalone.sh<\/code>. (This is a script included in the bin folder of the Apache Kafka zip you can download from kafka.apache.org).<\/p>\n
connect-standalone.sh connect.properties connector.properties<\/pre>\n
We left this running to create a live stream of events that we could use for demos.<\/p>\n<\/details>\n
\n\n                        We now had streams of events on MSK topics, ready to process using IBM Event Processing.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We started by following the Transform events to create or remove properties<\/strong> tutorial from ibm.github.io\/event-automation\/tutorials<\/a>.<\/p>\n
We were running an instance of IBM Event Processing<\/strong> in an OpenShift cluster in IBM Cloud. (For details of how we deployed this, you can see the first step in the tutorial instructions<\/a>. It just involved running an ansible playbook).<\/p>\n
We started by logging on to the Event Processing authoring dashboard.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
And we started to create a new flow.<\/p>\n<\/details>\n
\n\n                        We created an event source in Event Processing using an Amazon MSK topic.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We went to the MSK instance page, and clicked View client information<\/strong>. From there, we copied the public<\/strong> bootstrap address.<\/p>\n
We pasted that into the Server<\/strong> box in the event source configuration page. We needed to split up the comma-separated address we got from the Amazon MSK page, as Event Processing requires separate broker addresses.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We provided the consumer credentials we created earlier when setting up the MSK cluster.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The list of topics displayed in Event Processing matches the list of topics that we configured the consumer user to have access to.<\/p>\n
Following the tutorial instructions<\/a>, we chose the ORDERS.NEW topic.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We copied in a sample message from the ORDERS.NEW topic.<\/p>\n<\/details>\n
\n\n                        Finally, we created an event processing flow using the events from an MSK topic.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We continued to create the flow as described in the tutorial instructions<\/a>.<\/p>\n
All of the Event Processing tutorials<\/a> can be followed as written using the Amazon MSK cluster that we created.<\/p>\n<\/details>\n<\/details>\n<\/div>\n
\n
                \"using<\/a><\/p>\n
\n                Next, we demonstrated the value that Event Endpoint Management<\/strong> can bring to an Amazon MSK cluster. We showed how adding MSK topics to a self-service catalog enables sharing and reuse of existing topics, wherever they are hosted. And we showed the way that the addition of an Event Gateway can maintain control and governance of these topics as they are shared.<\/p>\n
                The diagram above is a simplified description of what we created. We used the same MSK cluster in AWS that we had used for the previous demo, as it already had a variety of topics and a data generator producing live streams of events to them. This time we used it with an instance of Event Endpoint Management (that was running in our Red Hat OpenShift cluster in IBM Cloud).<\/p>\n
We added our Amazon MSK topics to the catalog, and configured an Event Gateway to govern access to them. We could have run the Event Gateway in OpenShift, alongside the endpoint manager. However, for this demonstration, we wanted to show the flexibility of running the Event Gateway in the same environment as a Kafka cluster. This showed how you can remove the need for egress from the AWS environment where your Kafka applications are also running in AWS.<\/p>\n
Finally, we showed all of this in action by running a Kafka consumer, consuming events from the MSK topics. The consumer was using credentials created in the Event Endpoint Management catalog and connected via the Event Gateway.<\/p>\n
\n\n                    How we created a demonstration of this…
\n<\/summary>\n
\n\n                        We started by creating a security group (for the load balancer that would provide the external connection for the Event Gateway).
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We started by going to the Security Groups<\/strong> section of the EC2 service in AWS.<\/p>\n
We clicked Create security group<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We created a security group that would accept connections on port 443<\/code>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Next, we went to the IP Target groups<\/strong> page and clicked Create target group<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose IP addresses<\/strong> as the target type, and gave it a name that explained this would be the target address for the Event Gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose port 443<\/code>. Other port numbers would have been fine, but this is consistent with the port number used to access the Event Gateway when it is running in OpenShift.<\/p>\n
For the VPC<\/strong>, we chose the same VPC that the MSK cluster is in.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Next, we defined a healthcheck. Again, we did this to be consistent with the way the Event Gateway runs when managed by the Operator in OpenShift, by using the same protocol (HTTP<\/code>), port number (8081<\/code>), and path (\/ready<\/code>) that are used for probes in Kubernetes.<\/p>\n
Then we clicked Next<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The default values are fine for the next Register targets<\/strong> step, so we clicked Create target group<\/strong>.<\/p>\n<\/details>\n
\n\n                        Then, we created the network load balancer to give the Event Gateway an external address.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We started by going to the Load balancers<\/strong> page within EC2, and clicked Create load balancer<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose Network Load Balancer<\/strong> and clicked Create<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave it a name, and made it Internet-facing<\/strong> because we wanted the event gateway to be accessible from outside of AWS.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We selected one of the three availability zones created when we set up the MSK cluster to put the gateway in.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Then we selected security groups for the load balancer to use: the security group created for the MSK cluster, and the new one created for the gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
That was everything we needed, so we clicked Create load balancer<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
It took a little while to complete the setup.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We waited for it to no longer be in a Provisioning<\/strong> state before continuing.<\/p>\n<\/details>\n
\n\n                        Now that we had an external DNS address, we could create a certificate for the Event Gateway.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We clicked into the network load balancer that we’d just created to view the DNS name that we would give to the gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We’d installed IBM Event Endpoint Management using the demo ansible playbook<\/a>. This setup uses a Kubernetes Certificate Manager in OpenShift to setup the SSL\/TLS certs needed. This meant the issuer and CA we needed to create a certificate for the Event Gateway we would run in AWS was in OpenShift.<\/p>\n
Our next step was to go into OpenShift so that we could create a new cert.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We just needed to copy the DNS name<\/strong> for the gateway to use in the cert.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
That gave us what we needed to create the certificate.<\/p>\n
Notice that we used the DNS name in two places: spec.dnsNames<\/code> and spec.uris<\/code>. You can see more information about the requirements for this certificate in the Event Endpoint Management documentation<\/a>.<\/p>\n
apiVersion: cert-manager.io\/v1\nkind: Certificate\nmetadata:\n    name: amazon-eem-gw-cert\n    namespace: event-automation\nspec:\n    dnsNames:\n    - loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com\n    duration: 2160h0m0s\n    issuerRef:\n        kind: Issuer\n        name: my-eem-gateway-ibm-egw-iss\n    privateKey:\n        algorithm: RSA\n        rotationPolicy: Always\n    secretName: amazon-eem-gw-cert\n    subject:\n        organizations:\n        - IBM Event Endpoint Management\n    uris:\n    - egw:\/\/loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com:443\/amazon-gateways\/eu-west-1a\n    usages:\n    - client auth\n    - digital signature\n    - server auth<\/pre>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The cert manager created the certificate based on this spec, and stored it in a Kubernetes secret in the event-automation<\/code> namespace where Event Endpoint Management was running.<\/p>\n
We downloaded ca.crt<\/code>, tls.crt<\/code>, and tls.key<\/code> to three files that would be needed to run the gateway.<\/p>\n<\/details>\n
\n\n                        To make the certificates available to the Event Gateway we would run in AWS, we created a container image that would hold the certificates.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We decided to use a custom container image to hold the certificates that we’d just created. We wanted to store this as a private image in Amazon’s container registry, so we went to the ECR service in AWS.<\/p>\n
Then we clicked Get started<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose the repository name event-gateway-certs<\/code> and set the visibility to Private<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
That gave us an image name that we could push a container image to. We clicked on the View push commands<\/strong> button to get the details for this.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
First we needed to copy the login command.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used this to build and push a custom container image to ECR. Our Dockerfile<\/a> looked like this:<\/p>\n
FROM registry.access.redhat.com\/ubi8\/ubi-minimal:latest\n\nCOPY ca.crt \/certs\/eem\/ca.pem\nCOPY tls.crt \/certs\/eem\/client.pem\nCOPY tls.key \/certs\/eem\/client.key\nCOPY tls.crt \/certs\/eem\/egwclient.pem\nCOPY tls.key \/certs\/eem\/egwclient-key.pem\n\nVOLUME [\"\/certs\"]<\/pre>\n
The role of this container image would simply be to make the certificate files that we had downloaded in the previous step available to the Event Gateway.<\/p>\n
More information about the certificate file names we used, and the way these files would be used, can be found in the documentation for running stand-alone Event Gateways<\/a>.<\/p>\n<\/details>\n
\n\n                        We then created another security group, this time for the Event Gateway.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The previous security group we had created was for the network load balancer that would provide the external address for the Event Gateway. That was a security group that needed to allow connections from external clients.<\/p>\n
Now we needed to create a security group for the Event Gateway container itself, which would need to receive connections from the load balancer, and make connections to the Amazon MSK cluster.<\/p>\n
We started by going back to the Security Groups<\/strong> page.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave the security group a name that explained what this would be used for.<\/p>\n
We defined two inbound rules.<\/p>\n
The first was for the healthcheck address we had defined previously. This does not return sensitive data, so we allowed connections to this from anywhere, which would allow both the load balancer and ECS to use the healthcheck.<\/p>\n
The second was for the port used for Kafka traffic coming from the network load balancer, so we specified the 8092<\/code> port number and the security group for the network load balancer.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The security group was now ready to use.<\/p>\n<\/details>\n
\n\n                        We created a task definition for running the Event Gateway in ECS.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We went to the Elastic Container Service next, to start creating a new task definition for the Event Gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave our task definition the name loosehanger-event-gateway<\/code>.<\/p>\n
We opted for a launch type of AWS Fargate<\/strong> as that was a serverless option that would let us run the Event Gateway when it was being used.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose Linux\/X86\/64<\/strong> and a relatively small task size for our quick demo.<\/p>\n
We also needed to create a Task execution role<\/strong> to run the container. It needed to be able to pull the Event Gateway from the IBM Entitled Registry, so the task execution role needed to have access to credentials for this.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To enable this, we opened the AWS Secrets Manager<\/strong> so that we could Store a new secret<\/strong> to hold an Entitled Registry pull secret.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We created a new entitlement key from myibm.ibm.com\/products-services\/containerlibrary<\/a>.<\/p>\n
We used this to create a new Other type of secret<\/strong> using the Plaintext<\/strong> tab to create a secret like this:<\/p>\n
{\n    \"username\": \"cp\",\n    \"password\": \"our-entitled-registry-key\"\n}<\/pre>\n
Note that your entitled registry key is a password that you shouldn’t share publicly. The key you can see in the screenshot above has since been revoked.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We encrypted this secret using the encryption key that we’d created earlier.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave the secret a helpful description to remind us what this secret contains.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
The secret was ready to use. To use it in our task definition, we clicked into the instance to get the ARN for it.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We copied the ARN<\/strong> for the secret – so that we could add this to the container definition as the pull secret.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Back in the ECS task definition, we started filling in the details of the Event Gateway container.<\/p>\n
As described in the Event Endpoint Management documentation for installing a stand-alone gateway<\/a>, we needed to use an Image URI<\/strong> of cp.icr.io\/cp\/ibm-eventendpointmanagement\/egw:11.0.5<\/code><\/p>\n
We marked this as an Essential container<\/strong>, and added two port mappings: one for the healthcheck, and one for Kafka traffic.<\/p>\n
We pasted the pull secret ARN<\/strong> in for pulling the Event Gateway image.<\/p>\n
The ID of this isn’t sufficient for it to be used – the task execution role also needed permission to access the pull secret and decrypt it.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To do this, once the task execution role was created, we needed to attach some additional policies. Firstly, we needed to give it permission to pull our custom private certificates image from ECR.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
In order for the task execution role to decrypt the Entitled Registry pull secret, we needed to give it details of the loosehanger-keys<\/strong> encryption key we’d created earlier. We started by getting the ARN<\/strong> for this key.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
In order to pull the Event Gateway image from the Entitled Registry, the task execution role needs to be access:<\/p>\n
* the Entitled Registry we’d stored in Secrets Manager<\/p>\n
* the decryption key needed to be able to decrypt that secret<\/p>\n
We added the ARNs for both of these to a custom permissions policy.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Finally, we needed to set some environment variables to add to the Event Gateway container. To get the right values for this, we returned to the Event Endpoint Management manager in OpenShift.<\/p>\n
We needed to copy the API<\/strong> endpoint URI.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This provided the value for the backendURL<\/code> environment variable, which we set to https:\/\/my-eem-manager-ibm-eem-gateway-event-automation.itzroks-120000f8p4-ivahj5-6ccd7f378ae819553d37d5f2ee142bd6-0000.eu-gb.containers.appdomain.cloud<\/code><\/p>\n
We set the value of the GATEWAY_PORT<\/code> environment variable to 8092<\/code> as that is the port number we had chosen for Kafka traffic in the port mappings.<\/p>\n
We set the value of the GATEWAY_HEALTH_PORT<\/code> environment variable to 8081<\/code> as that is the port number we had chosen for the healthcheck in the port mappings.<\/p>\n
We set the value of the KAFKA_ADVERTISED_LISTENER<\/code> environment variable to loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com:443<\/code> as it was the DNS name we had created for the Event Gateway load balancer.<\/p>\n
We set the value of the certPaths<\/code> environment variable to this, as it matched the locations we’d created in the custom certs container we had built.<\/p>\n
\/certs\/eem\/client.pem,\/certs\/eem\/client.key,\/certs\/eem\/ca.pem,\/certs\/eem\/egwclient.pem,\/certs\/eem\/egwclient-key.pem<\/pre>\n
Finally, we added a few additional environment variables to match the way the gateway is run when managed by the Operator in OpenShift.<\/p>\n
GATEWAY_REGISTRATION_SPEC_LOCATION\n\/opt\/ibm\/gateway\/openapi-specs\/gw-director-openapi.yaml\n\nexternalGatewayConfigJsonFile\n\/config\/gatewayConfig.json\n\nfeatures\neemgw,core<\/pre>\n
These were all of the environment variables that we needed.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We enabled logging, as it can be useful when identifying what the gateway is doing.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We defined a healthcheck action (a curl<\/code> of the gateway’s readiness URL) for the container. Again, we were following the pattern for the probes that are set up when the Operator runs the Event Gateway in Kubernetes.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Finally, we needed to add the certificates we created for the gateway available by adding the custom certificates container we built as a secondary container.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave the certs container a command so that ECS wouldn’t treat it as a crashed container. We didn’t need this container to run anything, we just needed it to share a volume with the Event Gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We achieved this by giving the gateway container read-only access to the storage from the certificates container.<\/p>\n
This was everything we needed to complete the task specification, so at this point we clicked Create<\/strong>.<\/p>\n<\/details>\n
\n\n                        Now we were ready to create an ECS cluster to run our Event Gateway task.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We were ready to run the Event Gateway task, so we went to the Clusters<\/strong> page in ECS and clicked Create cluster<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We defined a Fargate<\/strong> cluster and gave it a name.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Then we waited it for it to be provisioned.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Once it was ready, we clicked into the cluster instance, went to the Services<\/strong> tab, and clicked the Create<\/strong> button.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
For the compute options, we opted for the Launch type<\/strong> compute option – as we only wanted to run a single instance of the gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose our Event Gateway task definition, and gave the service a name.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose the same availability zone to run the Event Gateway in that we’d chosen when creating the network load balancer.<\/p>\n
Then we chose the security group we had created for the gateway container.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We chose the load balancer we had created for the gateway.<\/p>\n
For the listener, we used the listener we had defined when creating the load balancer.<\/p>\n
Finally, we clicked Create<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Now we just needed to wait for the Event Gateway to start.<\/p>\n<\/details>\n
\n\n                        We verified that the Event Gateway running in AWS had connected to the Endpoint manager running in OpenShift.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
A good way to know when the Event Gateway is ready is to check the Gateways<\/strong> tab in the Event Endpoint Management manager. We could see our new Event Gateway listed in the table (alongside the existing Event Gateway we had created in OpenShift in IBM Cloud).<\/p>\n<\/details>\n
\n\n                        We could now start adding Amazon MSK topics to the Event Endpoint Management catalog.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
First, we went to the topics tab in the Event Endpoint Management manager.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to add the Amazon MSK cluster first, so we clicked Add new cluster<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave this new Kafka cluster a name: Amazon MSK<\/code>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to get the bootstrap address for the MSK cluster, using the same View client information<\/strong> button as before. Note that we needed to split the public bootstrap address we got from Amazon into separate broker addresses.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to accept the certificates when prompted.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used the consumer credentials that we had created for the MSK cluster, which had permission to consume from all of our application topics.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
With the credentials accepted, we were now ready to start adding topics.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We confirmed that all of the topics our consumer credentials had access to were listed.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We selected all of them. We could have given them different aliases, but we decided to use the existing topic names as-is.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Our topics were now added to the catalog, ready for additional documentation and publishing.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
For each topic, we needed to write additional documentation.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We added a recent message from the topic as a sample, to make the catalog documentation more useful.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Finally, we could click Publish<\/strong>.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to choose the gateway groups we wanted to be able to consume this topic through. We chose only the Event Gateway running in AWS.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Our topic was published! We were now able to repeat this for our other Amazon MSK topics.<\/p>\n<\/details>\n
\n\n                        Finally, to show this all working, we created a Kafka consumer to receive events from Amazon MSK topics through the Event Gateway.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
From the Event Endpoint Management catalog, we could review what people finding our topics in the catalog would see.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
To show this in action, we created credentials for consuming from the topic.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This gave us a unique username and password for consuming from this topic through the Event Gateway.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We clicked Download certificates<\/strong> to download the Event Gateway certificate as a PEM file.<\/p>\n
We copied the bootstrap address from the Servers<\/strong> section of the Catalog page.<\/p>\n
We put all of this: the location of the downloaded certificate file, the bootstrap address, and username and password we had generated from the Catalog, into a properties file. It looked like this:<\/p>\n
bootstrap.servers=loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com:443\nsecurity.protocol=SASL_SSL\nsasl.mechanism=PLAIN\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"eem-582bdccb-f1b4-479d-940a-7480147e70b1\" password=\"018c80b7-218b-4eed-a8f5-169d2cb64206\";\nssl.truststore.location=\/Users\/dalelane\/Downloads\/certificate_loosehanger-event-gateway-68367c28f6f6c440.elb.eu-west-1.amazonaws.com_443.pem\nssl.truststore.type=PEM\ngroup.id=test-consumer-group\nenable.auto.commit=false<\/pre>\n
We could use this from kafka-console-consumer.sh<\/code> to start consuming events through the Event Gateway.<\/p>\n<\/details>\n<\/details>\n<\/div>\n
\n
                \"using<\/a><\/p>\n
\n                Finally, we demonstrated how these could be combined<\/strong>, bringing the value of both the previous demonstrations together.<\/p>\n
                Making Amazon MSK topics available through a self-service catalog can enable much wider reuse of these streams of events. And providing a low-code authoring canvas for processing these events can extend this use beyond just developers, enabling both business and IT teams to define the scenarios they need to respond to.<\/p>\n
For this final demonstration, we again used the same Amazon MSK cluster, with the same range of topics and live streams of events as before. We had already added these to the Event Endpoint Management catalog for the previous demo, so for this demonstration we showed how MSK topics found in the catalog can easily be used in Event Processing to quickly identify new real-time insights.<\/p>\n
\n\n                    How we created a demonstration of this…
\n<\/summary>\n
\n\n                        We demonstrated an Event Endpoint Management catalog can contain topics from many distributions of Kafka cluster, including topics from Amazon MSK.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We had published two of the topics so far to be able to demonstrate Event Endpoint Management.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
With them both available in the catalog, that was enough to demonstrate another of the Event Processing tutorials.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
First, we started by showing that we could still add topics to the catalog from non-MSK Kafka clusters. We clicked Add topic<\/strong> and then chose an Event Streams<\/strong> cluster.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This was useful to show the benefits of aliases. If you have topics with the same name in different Kafka clusters, and want to add both to the catalog, aliases let you differentiate them.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
As the owner of the topic, we could see which topic(s) were hosted on which Kafka clusters.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
However, for someone finding the topics in the catalog, they all just look like Kafka topics. You can use tags to identify where the topics are from if you want developers to know, as we had done here.<\/p>\n<\/details>\n
\n\n                        Next, to process these topics using IBM Event Processing, we created a new flow.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We went back to the Event Processing UI, running in our OpenShift cluster in IBM Cloud.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We created a new event processing flow and gave it a name.<\/p>\n<\/details>\n
\n\n                        We defined event sources based on Amazon MSK topics, using details from the Event Endpoint Management catalog.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We created an event source, using the bootstrap address we copied from the Event Endpoint Management catalog page.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to accept the certificates as before.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used credentials that we created in the Event Endpoint Management catalog.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Then we confirmed the topic name.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We provided the sample message from the catalog to give Event Processing information about the properties of messages on this topic.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Then we started repeating this to add another event source.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
This second event source was also based on an Amazon MSK topic, that we discovered in the Event Endpoint Management catalog and accessed via the Event Gateway running in AWS.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Again, we provided the bootstrap address for the Event Gateway that we copied from the catalog page.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We needed to confirm the certificates.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used the Event Endpoint Management catalog to generate credentials unique for this second topic.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We added these credentials to the Event Processing configuration.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Then we confirmed the topic name.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
And finally provided the sample message to give Event Processing info about the properties of messages on this second topic.<\/p>\n<\/details>\n
\n\n                        Finally, we added an interval join to correlate events from these two MSK topics.
\n▶<\/span>
\n<\/summary>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used an Interval join<\/strong> to join these two MSK topics.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We gave this join a name.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used the assistant to help us identify the attributes of each stream of events that we could join on.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
We used the visualisation to help us specify the join window.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Then we chose, and named, the attributes from each stream of events that we wanted to output.<\/p>\n
\n
\n                            <\/a><\/div>\n
click on the image to enlarge<\/div>\n<\/div>\n
Finally, to show everything working, we ran the flow and viewed the results.<\/p>\n<\/details>\n<\/details>\n<\/div>\n
Our goal with this blog post was to demonstrate what can be done with IBM Event Automation<\/a>, with a particular focus on the benefits of composability. By taking advantage of the de-facto standard nature of the Kafka protocol, we can layer additional capabilities on top of Apache Kafka clusters, wherever they are running.<\/p>\n
Our demonstrations were intended to provide an illustrative example of using Event Automation with MSK. It was absolutely not meant to be a description of how to use Amazon services in a perfect or optimum way, but instead focused on a quick and simple way to show what is possible. We wanted to inspire you for how you could get more out of your own Amazon MSK cluster.<\/p>\n
For more information about any of the ideas that we have shared here, please see the Event Automation documentation<\/a>, or get in touch.<\/p>\n
\n
Amazon Web Services (AWS), Amazon Managed Streaming for Kafka (MSK), Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Container Registry (ECR) are trademarks of Amazon Web Services. This blog post is intended to explain how we demonstrated the use of these services, but does not imply any affiliation with, or endorsement by, Amazon or AWS. <\/small><\/p>\n","protected":false},"excerpt":{"rendered":"
Written with Chris Patmore IBM Event Automation helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have. A […]<\/p>\n","protected":false},"author":1,"featured_media":5026,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[593,611,584],"class_list":["post-5023","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ibm","tag-apachekafka","tag-eventautomation","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5023","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5023"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5023\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/5026"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5023"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5023"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5023"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}