{"id":5277,"date":"2024-09-11T12:30:57","date_gmt":"2024-09-11T12:30:57","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5277"},"modified":"2026-04-02T18:31:15","modified_gmt":"2026-04-02T18:31:15","slug":"using-ibm-event-endpoint-management-with-kafka-connect","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5277","title":{"rendered":"Using IBM Event Endpoint Management with Kafka Connect"},"content":{"rendered":"

You\u2019ve discovered a topic in the IBM Event Endpoint Management<\/a> catalog that someone in your company has shared. It looks useful, so you want to use that stream of events to maintain a local projection in your database.<\/strong><\/p>\n

Or maybe you\u2019ve discovered a topic in the Catalog that is available to produce to, and you want to contribute events to it from your MQ queue.<\/strong><\/p>\n

What are the options for using Kafka Connect to produce to, or consume from, topics that you discover in Event Endpoint Management?<\/strong><\/p>\n

<\/p>\n

In this post, we\u2019ll share options that you can consider, and briefly outline the pros and cons of each.<\/p>\n

Co-written with Andrew Borley<\/a><\/em><\/p>\n

<\/p>\n

Connect typically uses multiple Kafka topics<\/h3>\n

First, a quick reminder about how Kafka Connect works, and why this is a factor if you want to use it with Event Endpoint Management.<\/p>\n

A Kafka Connect runtime can support and manage a wide variety of connectors – bringing events from external systems into Kafka, or sending data from Kafka to external systems.<\/p>\n

<\/p>\n

In addition to those topics that individual connectors interact with, Connect also uses a few topics to store information:<\/p>\n

– one topic stores the configuration for the connectors it is managing
\n– another topic stores the current status of the connectors
\n– a third topic stores the progress that each connector has made so far to allow them to resume after a restart<\/p>\n

When you use Event Endpoint Management, you have the self-service opportunity to create individual credentials for a single, specific topic that you\u2019ve discovered from another team\u2019s Kafka cluster.<\/p>\n

But that doesn\u2019t let you use their Kafka cluster for your Connect runtime to also store it\u2019s config, status, and offsets.<\/p>\n

So what can you do instead?<\/p>\n

Option 1 – Using an existing Kafka Connect runtime<\/h3>\n

<\/p>\n

If you are already using Kafka Connect, running existing Connectors with your own Kafka cluster, then it is simple for you to continue using that.<\/p>\n

Your Connect runtime will continue to use your own Kafka cluster to store it\u2019s config, status, and offsets.<\/p>\n

apiVersion: eventstreams.ibm.com\/v1beta2\nkind: KafkaConnect\nmetadata:\n  annotations:\n    eventstreams.ibm.com\/use-connector-resources: 'true'\n  name: your-kafka-connect\nspec:\n  config:\n    group.id: your-kafka-connect\n    client.id: your-kafka-connect\n    config.storage.topic: connect-configs\n    offset.storage.topic: connect-offsets\n    status.storage.topic: connect-status\n  bootstrapServers: 'your-kafka-cluster-bootstrap:9092'\n  authentication:\n    type: scram-sha-512\n    username: your-kafka-cluster-credentials\n    passwordSecret:\n      password: password\n      secretName: your-kafka-cluster-credentials\n  tls:\n    trustedCertificates:\n      - certificate: ca.crt\n        secretName: your-kafka-cluster-ca-cert\n  externalConfiguration:\n    volumes:\n      - name: eem-cert\n        secret:\n          secretName: eem-gateway-ca-cert\n  ...\n<\/pre>\n
Most of your sink and source connectors will be consuming and producing using topics in your own Kafka cluster.<\/p>\n
Individual source connectors can be configured to produce to topics that you discover in Event Endpoint Management by using `producer.override.`.<\/p>\n
apiVersion: eventstreams.ibm.com\/v1beta2\nkind: KafkaConnector\nmetadata:\n  name: source-connector-to-eem-topic\n  labels:\n    eventstreams.ibm.com\/cluster: your-kafka-connect\nspec:\n  class: com.acme.externalsystem.SomeSourceConnector\n  config:\n    acme.externalsystem.source.config: config-values\n    ...\n    topic: EEM.PRODUCER.TOPIC\n    producer.override.sasl.mechanism: PLAIN\n    producer.override.sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username=\"eem-e2cb10d0-74ec-45ee-9741-c185b20059c3\" password=\"60b2dae1-d663-4a3c-8b1f-1e2475150616\";\n    producer.override.bootstrap.servers: 'eem-gateway-address:443'\n    producer.override.security.protocol: SASL_SSL\n    producer.override.ssl.truststore.location: \/opt\/kafka\/external-configuration\/eem-cert\/eem-truststore.jks\n    producer.override.ssl.truststore.type: JKS\n    producer.override.ssl.truststore.password: truststorepwd\n    producer.override.ssl.protocol: TLSv1.3\n<\/pre>\n
Individual sink connectors can be configured to consume from topics that you discover in Event Endpoint Management by using `consumer.override.`.<\/p>\n
apiVersion: eventstreams.ibm.com\/v1beta2\nkind: KafkaConnector\nmetadata:\n  name: sink-connector-from-eem-topic\n  labels:\n    eventstreams.ibm.com\/cluster: your-kafka-connect\nspec:\n  class: com.acme.externalsystem.SomeSinkConnector\n  config:\n    acme.externalsystem.sink.config: config-values\n    ...\n    topics: EEM.CONSUMER.TOPIC\n    consumer.override.sasl.mechanism: PLAIN\n    consumer.override.sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username=\"eem-e2cb10d0-74ec-45ee-9741-c185b20059c3\" password=\"60b2dae1-d663-4a3c-8b1f-1e2475150616\";\n    consumer.override.bootstrap.servers: 'eem-gateway-address:443'\n    consumer.override.security.protocol: SASL_SSL\n    consumer.override.ssl.truststore.location: \/opt\/kafka\/external-configuration\/eem-cert\/eem-truststore.jks\n    consumer.override.ssl.truststore.type: JKS\n    consumer.override.ssl.truststore.password: truststorepwd\n    consumer.override.ssl.protocol: TLSv1.3\n<\/pre>\n
Benefits<\/h4>\n
This approach is a good fit if you are already using your own Kafka cluster for your own applications and microservices, and are looking to use some additional select topics that you have found in the Event Endpoint Management Catalog.<\/p>\n
This is because you already have a Kafka cluster that you can use for Connect to store it\u2019s state in, and can just use the Event Gateway for individual select source and sink topics.<\/p>\n
Challenges<\/h4>\n
This approach is less appropriate if you don\u2019t have a Kafka cluster, as the overhead of creating a new Kafka cluster solely to host a few state topics for Connect is likely onerous.<\/p>\n
Option 2 – Using Kafka Connect in standalone mode<\/h3>\n
<\/p>\n
If you are not already running Kafka Connect, and are going to be deploying it solely for using topics that you discover in Event Endpoint Management, then it may be more appropriate to run Connect in standalone mode.<\/p>\n
In this mode, Kafka Connect uses its local filesystem to store config, status, and progress offsets in a file on the local disk.<\/p>\n
offset.storage.file.filename=\/location\/connect.state\nbootstrap.servers=eem-gateway-address:443\nsasl.mechanism=PLAIN\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"eem-e2cb10d0-74ec-45ee-9741-c185b20059c3\" password=\"60b2dae1-d663-4a3c-8b1f-1e2475150616\";\nsecurity.protocol=SASL_SSL\nssl.truststore.location=eem-truststore.jks\nssl.truststore.type=JKS\nssl.truststore.password=truststorepwd\nssl.protocol=TLSv1.3\nkey.converter=org.apache.kafka.connect.storage.StringConverter\nkey.converter.schemas.enable=false\nvalue.converter=org.apache.kafka.connect.storage.StringConverter\nvalue.converter.schemas.enable=false\n<\/pre>\n
Individual source or sink connectors can identify the topic that they want to use, and it will use the credentials from the Event Endpoint Management catalog that you have put in the Connect properties config.<\/p>\n
connector.class=com.acme.externalsystem.SomeSinkConnector\nacme.externalsystem.sink.config=config-values\ntopics=EEM.CONSUMER.TOPIC<\/pre>\n
connector.class=com.acme.externalsystem.SomeSourceConnector\nacme.externalsystem.source.config=config-values\ntopic=EEM.PRODUCER.TOPIC<\/pre>\n
Benefits<\/h4>\n
This approach is a good fit if you are don\u2019t have access to another Kafka cluster, other than the topics you find in Event Endpoint Management.<\/p>\n
Challenges<\/h4>\n
Running Connect in a standalone mode like this requires a little more effort. You will need to consider issues such as how to restart it if it crashes. You will need to ensure that the file used to store config and state is safe and highly available. All of these are things that are handled for you when running managed Kafka Connect clusters.<\/p>\n
This approach also restricts you from being able to distribute multiple Connect tasks across multiple workers, as you are running the whole connector within a single Java process. As a result, it may not be optimal for high-throughput connectors that would benefit from being parallelised across multiple workers.<\/p>\n
Option 3 – Using Mirror Maker 2 to maintain a local copy<\/h3>\n
<\/p>\n
This is a special-case variant of option 1, but is an interesting use case to call out.<\/p>\n
Consider that you’ve found a topic in the Event Endpoint Management catalog that you think looks useful or interesting for your own application(s).<\/p>\n
You could get your application(s) to consume directly from the Event Gateway, and that is the typical way people use topics from Event Endpoint Management.<\/p>\n
However, in some situations you may prefer to have your own local copy of the topic for your local applications to consume from.<\/p>\n
This can be achieved by running Mirror Maker 2, with the credentials and certificates that you obtain from Event Endpoint Management.<\/p>\n
kind: KafkaMirrorMaker2\napiVersion: eventstreams.ibm.com\/v1beta2\nmetadata:\n  name: mirror-eem\nspec:\n  # run Mirror Maker with the local mirror\n  connectCluster: local-mirror\n  clusters:\n    # the Event Endpoint Management gateway where a topic was found\n    - alias: event-endpoint-management\n      bootstrapServers: eem-gateway-address:443\n      authentication:\n        username: eem-7ea2c1b7-229d-4acd-b019-2a9e7326df65\n        passwordSecret:\n          password: password\n          secretName: eem-username\n        type: plain\n      tls:\n        trustedCertificates:\n          - certificate: eem-ca.pem\n            secretName: eem-ca-cert\n    # your local mirror where you want to maintain a copy of the EEM topic\n    - alias: local-mirror\n      bootstrapServers: local-mirror-kafka-bootstrap:9094\n      authentication:\n        username: local-mirror-user\n        passwordSecret:\n          password: password\n          secretName: local-mirror-user\n        type: scram-sha-512\n      tls:\n        trustedCertificates:\n          - certificate: ca.crt\n            secretName: local-mirror-cluster-ca-cert\n  mirrors:\n    - sourceCluster: event-endpoint-management\n      targetCluster: local-mirror\n      topicsPattern: EEM.CONSUMER.TOPIC\n      sourceConnector:\n        config:\n          # use topics on the local mirror to store offset mappings\n          offset-syncs.topic.location: target\n          offset-syncs.topic.replication.factor: 1\n\n          # only a single topic will be mirrored so no need to refresh list\n          refresh.topics.enabled: false\n\n          # don't try to mimic the ACLs provided by EEM\n          sync.topic.acls.enabled: false\n    ...<\/pre>\n
Benefits<\/h4>\n
Maintaining a local clone of a topic is beneficial if your network connection to the Event Gateway is slow and\/or expensive. This is particularly valuable if you intend to repeatedly re-consume events on the topic multiple times, and\/or from many different client applications.<\/p>\n
This pattern allows you to fetch the data from the remote cluster (through the Event Gateway) once only, which you can then repeatedly consume from locally.<\/p>\n
More generally, the idea here is that Mirror Maker can be used to maintain a local clone of a topic that you discover in the Catalog, that you are then free to do with as you wish.<\/p>\n
Challenges<\/h4>\n
As with Option 1 above, the cost of this pattern is that you need to run your own local Kafka cluster, however if you already have this, then it is worth considering.<\/p>\n","protected":false},"excerpt":{"rendered":"
Options for using Kafka Connect with topics that you discover in IBM Event Endpoint Management<\/p>\n","protected":false},"author":1,"featured_media":5278,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[593,584],"class_list":["post-5277","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ibm","tag-apachekafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5277"}],"version-history":[{"count":2,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5277\/revisions"}],"predecessor-version":[{"id":5974,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5277\/revisions\/5974"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/5278"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}