{"id":5223,"date":"2024-04-26T17:38:01","date_gmt":"2024-04-26T17:38:01","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5223"},"modified":"2026-04-02T17:40:58","modified_gmt":"2026-04-02T17:40:58","slug":"using-ibm-event-automation-with-azure-event-hubs","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5223","title":{"rendered":"Using IBM Event Automation with Azure Event Hubs"},"content":{"rendered":"<p><a href=\"https:\/\/www.ibm.com\/products\/event-automation\">IBM Event Automation<\/a> helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have.<\/p>\n<p>A key goal of Event Automation is to be composable. The three components can be used together, or they can each be used to extend and enhance an existing event-driven deployment.<\/p>\n<p>Today, I demonstrated some of the Event Automation components working with <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/event-hubs\/azure-event-hubs-kafka-overview\">Azure Event Hubs for Apache Kafka<\/a>. As Event Hubs provides a Kafka interface to Azure&#8217;s data streaming service, it obviously can be used with Event Automation. But it can be helpful to inspire people by showing it for real, so even demos of obvious things can be valuable.<\/p>\n<p>For example, Event Endpoint Management can enhance the value of topics in Event Hubs by offering management and governance, and by enabling governed reuse of those topics. Event Processing makes it easy to get insights from the events on Event Hubs topics, by providing an intuitive low-code authoring canvas to process them.<\/p>\n<p>If I was going to be running this for a while and wanted to optimise for my applications in Azure, I would likely have set this up like this, with the Event Gateways deployed close to the Azure Kafka endpoints.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/azure-complete.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p><!--more-->This is similar to how I deployed <a href=\"https:\/\/dalelane.co.uk\/blog\/?p=5023\">Event Automation with Amazon MSK<\/a> last year. If you&#8217;d like to know more about that deployment pattern, you can get all the details in that post.<\/p>\n<p>Instead, as my objective today was just to get people thinking about what could be possible, I went with this quicker and simpler approach:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/azure-simple.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>I captured screenshots of the initial set-up which I&#8217;m sharing here, to help anyone else who wants to recreate this sort of demo:<\/p>\n<ul>\n<li><a href=\"#step1\">Creating the Azure Event Hubs demo environment<\/a><\/li>\n<li><a href=\"#step2\">Prepare a demo Event Hub topic with demo messages<\/a><\/li>\n<li><a href=\"#step3\">Add the demo topic to <strong>Event Endpoint Management<\/strong><\/a><\/li>\n<li><a href=\"#step4\">Processing Azure Event Hubs topics in <strong>Event Processing<\/strong><\/a><\/li>\n<\/ul>\n<h3 id=\"step1\">Creating the Azure Event Hubs demo environment<\/h3>\n<p>Go to the Azure services portal and search for Event Hubs.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-02.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Click into the Event Hubs marketplace page and click <strong>Create<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-03.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Create a new resource group and namespace for the demo instance.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-04.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Note that you need to choose the standard, premium, or dedicated pricing tier, as <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/event-hubs\/azure-event-hubs-kafka-overview\">those are the tiers that include the Kafka interface<\/a>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-05.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>All the other options can be left to the default settings.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-06.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-07.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-08.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Click <strong>Create<\/strong> to start creating the instance.<\/p>\n<p>It takes a brief moment for the deployment to complete.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-09.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Once the deployment is complete, click into the resource to start setting it up.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-10.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Confirm that <strong>Kafka surface<\/strong> is shown as <strong>ENABLED<\/strong>, which it should be as long as you selected a pricing tier that supports the Kafka interface.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-11.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<h3 id=\"step2\">Prepare a demo Event Hub topic with demo messages<\/h3>\n<p>Click on <strong>+ Event Hub<\/strong> to create a new topic.<\/p>\n<p>I gave my topic the name <code>orders<\/code>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-12.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Otherwise, the default settings are fine.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-13.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Click <strong>Create<\/strong> and wait for the new topic to be created.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-14.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>The next thing to do is to put some messages on the new topic that can be used for testing.<\/p>\n<p>Click on <strong>Shared access policies<\/strong>, then click <strong>Add<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-15.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Name the credentials to use to put some test messages on the topic, and then click <strong>Create<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-16.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>The <strong>Connection string-primary key<\/strong> is the password to use for a Kafka client.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-17.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>The standard Kafka console producer is sufficient to produce a small number of test messages useful for a demo.<\/p>\n<p>The boostrap servers value to use is the host name from the Azure Event Hub overview page, followed by <code>:9093<\/code>.<\/p>\n<p>The username to use is <code>$ConnectionString<\/code>.<\/p>\n<p>The password to use is the <strong>Connection string-primary key<\/strong> value from the policy.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-18.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<h3 id=\"step3\">Add the demo topic to Event Endpoint Management<\/h3>\n<p>The starting point is the <strong>Topics<\/strong> page in the Event Endpoint Management UI.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-20.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Adding the Azure Event Hubs topic alongside topics from other Kafka distributions is a nice way to illustrate the value of bringing topics from multiple Kafka clusters into a single catalog.<\/p>\n<p>Click on <strong>Add topic<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-21.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>For this demo, I wanted to consume the test messages that I produced directly to the topic, so I chose <strong>Consume events<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-22.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Give the cluster a name.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-23.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>The server address to use is the host name from the Azure Event Hub overview page, followed by <code>:9093<\/code>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-24.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>From the Azure portal, create a new policy for use by the Event Endpoint Management gateway. I called it <strong>event-endpoint-management<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-19.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Copy the <strong>Connection string-primary key<\/strong> value as the password that the Event Gateway can use to access Azure Event Hubs topics.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-26.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>As before, the username is <code>$ConnectionString<\/code>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-27.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>This adds <strong>Azure Event Hubs<\/strong> to the list of clusters available in Event Endpoint Management.<\/p>\n<p>Click <strong>Next<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-28.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Choose the topic to add to the catalog.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-29.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Documenting the topic helps to enable greater reuse. In this demo, I used the name to highlight that this topic was from Azure Event Hubs, but I equally could have used tags to do this instead.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-30.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>For this demo, I had produced JSON messages, so I selected <code>application\/json<\/code> for the encoding.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-31.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>I copied one of the messages I produced earlier to use as a sample message in the catalog page.<\/p>\n<p>Click <strong>Save<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-32.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>The next step after describing the topic is to publish it to the catalog.<\/p>\n<p>Click on <strong>Options<\/strong>, then click on <strong>Create option<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-32.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Choose the topic name that applications can use when consuming from this topic through Event Endpoint Management. As I already have &#8220;orders&#8221; topics from other Kafka clusters in the catalog, I gave this topic the alias <code>ORDERS.AZURE<\/code>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-35.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Click <strong>Next<\/strong> and then click <strong>Publish<\/strong>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-36.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>The Azure Event Hubs topic is now in the catalog, alongside the other topics from my Event Streams clusters.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-37.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<h3 id=\"step4\">Processing Azure Event Hubs topics in Event Processing<\/h3>\n<p>The final step was to show that these events can be processed using Flink jobs created in Event Processing&#8217;s low-code UI.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-39.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>The server address for the Event Gateway can be copied from the catalog page for the Azure topic.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-38.png\" style=\"width: 100%; max-width: 800px; \"\/><\/p>\n<p>Paste the gateway address into the server field for the event source node.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-40.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>Create credentials for accessing the Azure Event Hubs topic from the Event Endpoint Management catalog page.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-41.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>Paste the credentials into the event source configuration in Event Processing.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-42.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>The messages for this demo are JSON strings, so choose <code>JSON<\/code> as the message format to use for processing.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-43.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>Copy the sample message from the Event Endpoint Management catalog page, and use that to configure the Event Processing event source.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-44.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>The sample message is used to generate a table structure for the events that will be consumed from Azure Event Hubs. Click <strong>Configure<\/strong> to confirm.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-45.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>To illustrate processing the messages, you can quickly use something like a <strong>Filter<\/strong> node.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-46.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-47.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>This is a trivially simple example of the processing that can be done, but now that the events are available, it&#8217;s an easy jumping off point to do all of the other kinds of processing that you can see <a href=\"https:\/\/ibm.github.io\/event-automation\/tutorials\/\">described in our tutorials<\/a>.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.dalelane.co.uk\/2024-04-26-azure\/screenshot-48.png\" style=\"width: 100%; max-width: 800px; border: thin black solid;\"\/><\/p>\n<p>The aim of doing this demo was to inspire some ideas of the ways that you can get more from Azure Event Hubs, by making it easy to:<\/p>\n<ul>\n<li>share and manage &#8211; using Event Endpoint Management<\/li>\n<li>process &#8211; using Event Processing<\/li>\n<\/ul>\n<p>For more information about any of this, please see the <a href=\"https:\/\/ibm.github.io\/event-automation\/\">Event Automation documentation<\/a>, or get in touch.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>IBM Event Automation helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have. A key goal of Event [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":5224,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[593,584],"class_list":["post-5223","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech","tag-apachekafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5223","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5223"}],"version-history":[{"count":1,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5223\/revisions"}],"predecessor-version":[{"id":5963,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5223\/revisions\/5963"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/5224"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}