{"id":5270,"date":"2024-06-30T11:52:10","date_gmt":"2024-06-30T11:52:10","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5270"},"modified":"2026-04-02T17:16:34","modified_gmt":"2026-04-02T17:16:34","slug":"event-gateway-topologies-for-ibm-event-endpoint-management","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5270","title":{"rendered":"Event Gateway topologies for IBM Event Endpoint Management"},"content":{"rendered":"

In this post, I share a few examples for how to run Event Gateways for Event Endpoint Management.<\/strong><\/p>\n

When we talk about Event Endpoint Management<\/a>, we often draw logical diagrams like this, with Kafka client applications able to produce and consume events to back-end Kafka clusters via an Event Gateway.<\/p>\n

<\/a><\/p>\n

When it comes to start planning a deployment, we need to make decisions about the best way to create that logical Event Gateway layer. This typically includes running multiple gateways, but there are many different ways to do this, depending on your requirements for scaling and availability.<\/p>\n

For this post, I want to show two approaches for running two Event Gateways, as a way of illustrating the kind of topologies that are possible.<\/p>\n

<\/p>\n

Example 1 – more gateways<\/h3>\n

When creating an EventGateway<\/code> instance, the replicas<\/code> property can be used to request multiple instances of the gateway.<\/p>\n

apiVersion: events.ibm.com\/v1beta1\nkind: EventGateway\nmetadata:\n  name: my-eem-gateway\n  namespace: environment-one\nspec:\n  gatewayGroupName: gateway-group\n  gatewayID: gateway-1\n  replicas: 2\n  managerEndpoint: 'https:\/\/my-eem-manager-ibm-eem-gateway-event-automation.apps.dale-lane.cp.fyre.ibm.com'\n  tls:\n    caSecretName: my-eem-manager-ibm-eem-manager-ca\n  license:\n    accept: true\n    license: L-HRZF-DWHH7A\n    metric: VIRTUAL_PROCESSOR_CORE\n    use: EventAutomationNonProduction<\/pre>\n
This will result in the Operator deploying multiple pods which will collectively provide the Gateway service.<\/p>\n
% oc get pods --all-namespaces -l \"app.kubernetes.io\/instance=my-eem-gateway\"\nNAMESPACE         NAME                                            READY   STATUS    RESTARTS   AGE\nenvironment-one   my-eem-gateway-ibm-egw-gateway-5464fb95-mx5zr   1\/1     Running   0          29m\nenvironment-one   my-eem-gateway-ibm-egw-gateway-5464fb95-wk4k6   1\/1     Running   0          29m<\/pre>\n
This will be displayed in the Event Endpoint Manager catalog as a single address.<\/p>\n
<\/a>
Click on the image for a larger version<\/em><\/small><\/p>\n
At a simplified level, this is what is created as a result. The routes configured to front the gateway pods are configured to route requests to both \/ either of the replicas.<\/p>\n
(Routes are what are created when running in Red Hat OpenShift, but the same can be achieved using ingress on other Kubernetes distributions.<\/em>)<\/p>\n
<\/a><\/p>\n
When you run a Kafka consumer or producer, you configure it with bootstrap.servers<\/code> set to the single bootstrap address being used by both the gateway replicas.<\/p>\n
bootstrap.servers=my-eem-gateway-ibm-egw-rt-environment-one.apps.dale-lane.cp.fyre.ibm.com:443<\/pre>\n
The Route<\/code> will route the connection to one of the gateway pods, which will proxy the connection to the back-end Kafka cluster.<\/p>\n
One benefit of doing this could be scaling, with multiple gateway pods collectively handling requests to the gateway routes.<\/p>\n
<\/a><\/p>\n
Another benefit is availability. It is useful if anything happens to cause the gateway pod (that is handling the connection for a client application) to become unavailable. This could be due to an unexpected error, or simply due to a scheduled restart to apply maintenance.<\/p>\n
In such circumstances, Kafka client applications will automatically attempt to reconnect, and the same Routes<\/code> will automatically route the connection to the remaining available gateway pod.<\/p>\n
<\/a><\/p>\n
This means applications continue running without needing any administrative intervention.<\/p>\n
Example 2 – gateways in more locations<\/h3>\n
Alternatively, the two gateways can each be created in different environments.<\/p>\n
(For this simple example, I’m using Kubernetes namespaces to represent different environments, but this would work in the same way if I had used different Kubernetes clusters running in different availability zones.<\/em>).<\/p>\n
This can be achieved by creating two EventGateway<\/code> instances, each with a unique gatewayID<\/code>.<\/p>\n
Specifying a common gatewayGroupName<\/code> value indicates that the two gateway pods are part of the same logical group.<\/p>\n
apiVersion: events.ibm.com\/v1beta1\nkind: EventGateway\nmetadata:\n  name: my-eem-gateway\n  namespace: environment-one\nspec:\n  gatewayGroupName: gateway-group\n  gatewayID: gateway-1\n  replicas: 1\n  managerEndpoint: 'https:\/\/my-eem-manager-ibm-eem-gateway-event-automation.apps.dale-lane.cp.fyre.ibm.com'\n  tls:\n    caSecretName: my-eem-manager-ibm-eem-manager-ca\n  license:\n    accept: true\n    license: L-HRZF-DWHH7A\n    metric: VIRTUAL_PROCESSOR_CORE\n    use: EventAutomationNonProduction<\/pre>\n
apiVersion: events.ibm.com\/v1beta1\nkind: EventGateway\nmetadata:\n  name: my-eem-gateway\n  namespace: environment-two\nspec:\n  gatewayGroupName: gateway-group\n  gatewayID: gateway-2\n  replicas: 1\n  managerEndpoint: 'https:\/\/my-eem-manager-ibm-eem-gateway-event-automation.apps.dale-lane.cp.fyre.ibm.com'\n  tls:\n    caSecretName: my-eem-manager-ibm-eem-manager-ca\n  license:\n    accept: true\n    license: L-HRZF-DWHH7A\n    metric: VIRTUAL_PROCESSOR_CORE\n    use: EventAutomationNonProduction<\/pre>\n
This results in the Operator deploying a gateway pod in each of the EventGateway<\/code> locations.<\/p>\n
(Again, I’m using namespaces to represent environments – if doing this for real, it would more likely be different clusters<\/em>.)<\/p>\n
% oc get pods --all-namespaces -l \"app.kubernetes.io\/instance=my-eem-gateway\"\nNAMESPACE         NAME                                             READY   STATUS    RESTARTS   AGE\nenvironment-one   my-eem-gateway-ibm-egw-gateway-9b7458c5f-pqjpl   1\/1     Running   0          3m1s\nenvironment-two   my-eem-gateway-ibm-egw-gateway-7f954b486-cxsrj   1\/1     Running   0          3m10s<\/pre>\n
This will be displayed in the Event Endpoint manager catalog as separate addresses, one for each environment.<\/p>\n
<\/a>
Click on the image for a larger version<\/em><\/small><\/p>\n
At a simplified level, this is what is created as a result. Each gateway deployment has it’s own collection of Routes<\/code> fronting the connections to the gateway pod.<\/p>\n
<\/a><\/p>\n
When you run a Kafka consumer or producer, you configure it with bootstrap.servers<\/code> set to a comma-separated list of both of the gateway addresses.<\/p>\n
bootstrap.servers=my-eem-gateway-ibm-egw-rt-environment-one.apps.dale-lane.cp.fyre.ibm.com:443,my-eem-gateway-ibm-egw-rt-environment-two.apps.dale-lane.cp.fyre.ibm.com:443<\/pre>\n
Kafka clients will attempt to connect to the first address in the comma-separated list. If that connection is unsuccessful, it will retry with the next address in the list. This continues until it has made a successful connection, or it has tried every address in the list.<\/p>\n
This normal Kafka behaviour means you can choose how to use the two gateway environments.<\/p>\n
Configuring the Kafka application to randomize the order of addresses in the bootstrap.servers<\/code> list means it will randomize which environment the application uses to connect to the Kafka cluster via.<\/p>\n
Configuring the Kafka application with bootstrap.servers<\/code> set to environment-one,environment-two<\/code> as above means that all connections will be made to the gateway running in environment one.<\/p>\n
<\/a><\/p>\n
If anything happens to make the gateway in environment one unavailable (whether due to an unexpected or scheduled restart), Kafka client applications will automatically reconnect to the next address in the list, and start using the gateway in environment two.<\/p>\n
<\/a><\/p>\n
This means applications continue running without needing any administrative intervention.<\/p>\n
Example 3 – more gateways in more locations<\/h3>\n
These are just two examples of several topologies that can be created. For example, combining the two approaches described above would result in something like this – running Event Gateways in multiple environments, with multiple replicas in each environment.<\/p>\n
<\/a><\/p>\n
The ideal topology for any project will depend on the availability and throughput requirements for that project, but these examples will hopefully show you the kinds of options available.<\/p>\n","protected":false},"excerpt":{"rendered":"
In this post, I share a few examples for how to run Event Gateways for Event Endpoint Management. When we talk about Event Endpoint Management, we often draw logical diagrams like this, with Kafka client applications able to produce and consume events to back-end Kafka clusters via an Event Gateway. When it comes to start […]<\/p>\n","protected":false},"author":1,"featured_media":5271,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,584],"class_list":["post-5270","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-apachekafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5270","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5270"}],"version-history":[{"count":2,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5270\/revisions"}],"predecessor-version":[{"id":5945,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5270\/revisions\/5945"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/5271"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5270"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5270"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5270"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}