Configuring IBM App Connect Enterprise to consume messages from Kafka topics in IBM Event Endpoint Management requires careful configuration. In this post, I’ll share the steps I use that help me to avoid missing any required values.
If this sounds familiar, it might be because I wrote a post like this about using App Connect Enterprise to work with topics from Event Streams. People seem to have found that post useful, so I thought I’d do something similar for topics in Event Endpoint Management this time.
To illustrate this, I’ll create a simple App Connect flow that consumes messages from a Kafka topic and publish them to an MQTT topic.
The key to getting this to work correctly first time is to make sure that values are accurately copied from Event Endpoint Management to App Connect.
To help with this, I use a grid like the one below.
The instructions in this post start with Event Endpoint Management, and explain how to populate the grid with the information you need.
Then the instructions will switch to App Connect, and explain how to use the values in the grid to set up your App Connect flow.
| What this is | Values you will see in my screenshots | Your value | |
|---|---|---|---|
| A | Topic name |
DEMO.ACE
|
|
| B | Bootstrap address |
my-eem-gateway-ibm-egw-rt-event-automation.apps.dalelane.cp.fyre.ibm.com:443
|
|
| C | SASL mechanism |
PLAIN
|
|
| D | SASL config |
org.apache.kafka.common.security.plain.PlainLoginModule required;
|
|
| E | Security protocol |
SASL_SSL
|
|
| F | Certificate |
eem-cert.jks
|
|
| G | Certificate password |
STOREPASSW0RD
|
|
| H | Username |
eem-9c8fc5d9-fddd-48dd-ab41-e062214166e5
|
|
| I | Password |
dd08a1fc-99be-4931-8059-70aef88c1f0c
|
|
| J | Policy project name |
demo-policies
|
|
| K | Policy name |
demo-eem-policy
|
|
| L | Security identity name |
eem-credentials
|
|
| M | Truststore identity name |
eem-truststore
|
Note: To see screenshots in more detail, you can click on them to open a higher-resolution version.
Pre-requisites
You have found a topic in Event Endpoint Management that you want to use App Connect to consume messages from.
Before you start, to enable compatibility with the Kafka consumer in App Connect Enterprise, you will need to enable support for TLSv1.2 in the Event Endpoint Management gateway.
You can do this by adding an environment variable to your EventGateway custom resource.
env: - name: TLS_VERSIONS value: 'TLSv1.2,TLSv1.3'
Step 1
Collecting values you need from Event Endpoint Management
A – the topic name
Get the name of the topic you want App Connect to consume messages from.
Fill this in the grid as value A.
B – the bootstrap address
Get the bootstrap address of the Event Endpoint Management gateway that App Connect will use to connect to.
Click on the copy button next to the Servers address.
Fill the copied value in the grid as value B.
C – security mechanism
The Event Endpoint Management gateway supports SASL/PLAIN.
Fill PLAIN in the grid as value C.
D – security config
The Event Endpoint Management gateway supports SASL/PLAIN.
Fill org.apache.kafka.common.security.plain.PlainLoginModule required; in the grid as value D.
E – security protocol
The Event Endpoint Management gateway supports SASL/PLAIN.
Fill SASL_SSL in the grid as value E.
F – SSL certificate
The Event Endpoint Management gateway requires encrypted connections, so you need to download the CA certificate for the listener.
Download the PEM certificate from the Event Endpoint Management catalog.
App Connect Enterprise doesn’t support PEM files, so you need to convert this downloaded file into one of the supported formats, such as JKS.
For example:
keytool -importcert -noprompt \
-alias "eem-ca" \
-file your-downloaded-cert.pem \
-keystore eem-cert.jks \
-storetype JKS -storepass STOREPASSW0RD
Keep this file safe, and make a note of the file name in the grid as value F.
Fill in the password you use for the truststore (which I have as STOREPASSW0RD in my example command above) in the grid as value G.
H/I – username / password
Click on the Generate access credentials button.
Once you’ve provided your contact details, a new username and password will be generated for you.
Enter the generated username into the grid as value H.
Enter the generated password into the grid as value I.
Step 2
Choose some App Connect names
You will create some resources in App Connect.
Choose a name for your App Connect policy project.
Enter it into the grid as value J.
Choose a name for your App Connect policy.
Enter it into the grid as value K.
Choose a name for the security identity you will use to store the username and password.
Enter it into the grid as value L.
Choose a name for the security identity you will use to store the truststore password.
Enter it into the grid as value M.
Step 3
Creating your App Connect policy
Use the App Connect Enterprise toolkit to create a new Policy project.
Use the name from value J in the grid for the name.
Create a policy in your new project. Use value K from the grid as the file name.
Make sure that the policy name matches value K from the grid.
Set the policy Type and Template both to “Kafka”.
Fill in the rest of the policy using values from the grid.
Set Bootstrap servers (<bootstrapServers>) to value B from the grid.
Set Security protocol (<securityProtocol>) to value E from the grid.
Set SASL Mechanism (<saslMechanism>) to value C from the grid.
Set SSL protocol (<sslProtocol>) to TLSv1.2.
Set Security identity (<securityIdentity>) to value L from the grid.
Set SASL config (<saslConfig>) to value D from the grid.
Set SSL truststore location (<sslTruststoreLocation>) to /home/aceuser/truststores/ followed by your filename.
(e.g. /home/aceuser/truststores/eem-cert.jks).
Set SSL truststore type (<sslTruststoreType>) to JKS (if you created a jks file) or PKCS12 (if you created a p12 file).
Set SSL truststore security identity
(<sslTruststoreSecurityIdentity>) to value M from the grid.
Set SSL certificate hostname checking
(<sslEnableCertificateHostnameChecking>) to true.
It should look something like this:
Export the policy project to a zip file.
The file name you use isn’t significant, so choose any name that you like.
Step 4
Creating your App Connect flow
To illustrate how to use the grid, I created a flow that sends data received from the Kafka topic to an MQTT topic.
Start by creating a KafkaConsumer node.
Configure the Kafka node, starting with the “Basic” tab.
Set Topic name to value A from the grid.
The Bootstrap servers value won’t be used, but it’s a required value, so put any value in there. I use “not-used” for this to avoid confusion.
Next, fill in the “Security” tab.
Set Security identity to value L from the grid.
Set Security protocol to value E from the grid.
Set SSL protocol to TLSv1.2.
Finally, fill in the “Policy” tab.
Set Policy using value J and value K from the grid. It should look like {valueJ}:valueK.
For this demo, I’m publishing the messages received from the Event Endpoint Management topic to an MQTT topic.
Export the app with your flow to a BAR file.
Step 5
Set up App Connect
Create an App Connect dashboard
This will make it easier to deploy your App Connect flow.
Add the Configurations to the dashboard
Use the Dashboard to create Configurations.
Create a new Configuration.
Set the Type to “Truststore” and upload your truststore file.
Create another new Configuration.
Set the Type to “Policy project” and upload your exported policy project zip file.
Create another new Configuration.
Set the Type to “setdbparms.txt”.
Add a line with kafka:: followed by the security identity name for the credentials, then a space, then the username and password.
kafka::valueL valueH valueI
Add a line with truststore:: followed by the security identity name for the truststore, then a space, then an unused placeholder value, then the truststore password.
truststore::valueM notused valueG
You should now have three configurations.
Upload the BAR file to the dashboard
Use the Dashboard to upload a bar file.
Import the bar file that you created with the Kafka message flow.
Step 6
Deploy the message flow
Use the Dashboard to create a new integration server.
Note that creating an integration server with CPU and memory limits that are too small can result in SSL handshake errors when connecting to Kafka. If this happens, try different CPU and memory limits. For example, setting CPU limit to at least 1 and memory to at least 512Mi may help.
Choose your BAR file
Enable all of the Configurations that you created
Name the integration and click Create.
Step 7 – Try it out!
Your App Connect flow should now be consuming messages from the Kafka topic you discovered in Event Endpoint Management.
I verified my flow by creating a simple MQTT subscriber – I immediately saw events from the Kafka topic start to show up.
(If you want to know more about the MQTT app I’m using here, see my post on creating an MQTT extension for Scratch.)
Finished! Using a grid like the one detailed here should hopefully ensure that you have it working first time, without a lot of time-consuming errors and debugging.
Tags: apachekafka, appconnect, kafka