Setting up trusted SSL for IBM Event Streams

A quick how-to for setting up Event Streams with trusted certificates when running a development project.


You’re working on a project using IBM Event Streams. It’s just a development project, so you’re not using an SSL certificate signed by your real, trusted, corporate signer.

Everything works, but…

You get errors like these every time you access the web tooling – which you have to click through.

And you get errors like these from your Kafka client applications – which you have to configure with a custom truststore to avoid (although, if you do need to do that, I have a guide to help!)

[2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 ( failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process:  ($)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: PKIX path building failed: unable to find valid certification path to requested target
	at java.base/
	at java.base/
	at java.base/
	at java.base/
	at java.base/$T13CertificateConsumer.checkServerCerts(
	at java.base/$T13CertificateConsumer.onConsumeCertificate(
	at java.base/$T13CertificateConsumer.consume(
	at java.base/
	at java.base/
	at java.base/$DelegatedTask$
	at java.base/$DelegatedTask$
	at java.base/
	at java.base/$


The reason for these errors is because by default, Cloud Pak for Integration – in the absence of a certificate you’d rather use – has generated new self-signed certificates to use.

Your browser and client applications don’t trust this new self-signed certificate by default.


TLDR: Use a free certificate from Let’s Encrypt. When you set up Event Streams, configure it to use that certificate. That will (almost certainly) be trusted by all of your apps, and you won’t have to go through the annoying workarounds.

The rest of this post is just a guided walkthrough explaining how to do that.

To keep things simple, I’m going to assume that you’re starting from scratch – you have a brand new, vanilla, OpenShift cluster.


Some of the commands in this post use files that you can find at

Commands you should run will be highlighted like this
Commands to see what is happening will be shown like this

Step 1
Get a certificate


If you are using a managed OpenShift cluster in IBM Cloud (“ROKS”), a certificate is automatically created for you when you create the cluster. It’s already available as a Secret in the openshift-ingress namespace, and you can use that.

SECRET_NAME=$(oc get secret -n openshift-ingress  -o=jsonpath='{.items[?(@.metadata.annotations.ingress\.cloud\.ibm\.com/cert-source=="ibm")]}')

echo "Let's Encrypt certificate is available in the $SECRET_NAME secret in the openshift-ingress namespace"

You can download this to a few files on your computer.

First, the key:

oc get secret -n openshift-ingress $SECRET_NAME -o=jsonpath="{.data['tls\.key']}" | base64 -d > tls.key

Then the full chain, which you’ll need to split into the certificate and the CA’s.

oc get secret -n openshift-ingress $SECRET_NAME -o=jsonpath="{.data['tls\.crt']}" | base64 -d > full-chain.crt

split -p "-----BEGIN CERTIFICATE-----" full-chain.crt cert-

mv cert-aa tls.crt

cat cert-ab cert-ac > ca.crt

rm cert-ab cert-ac

You should now have three files:
– tls.key
– tls.crt
– ca.crt


If you’re running OpenShift somewhere else, you will need to create the certificate yourself. See the documentation for more guidance.

Step 2
Install the Cloud Pak for Integration operators

Add the IBM software catalog to your OpenShift cluster.

oc apply -f 01-ibm-catalog-source.yaml

Wait for the operator catalog to be updated with the IBM operators, then install the Cloud Pak for Integration operator.

oc apply -f 02-operator-platform-navigator.yaml

Wait for the IBM Cloud Pak for Integration operator and it’s pre-reqs (IBM Cloud Pak foundational services, and IBM NamespaceScope Operator) finish installing.

Step 3
Prepare the namespace

I’m going to use a namespace called integration in this walkthrough. You can obviously change this to anything you’d like in all the subsequent commands.

Create the namespace.

oc new-project integration

Create an entitled registry key in the namespace, that you’ll need to deploy IBM software.

oc create secret docker-registry ibm-entitlement-key \
    --docker-username=cp \
    --docker-password=$IBM_ENTITLEMENT_KEY \ \
    --namespace=integration --dry-run=client -o yaml | oc apply -f -

Step 4
Start creating Cloud Pak Foundational Services

Cloud Pak Foundational Services provides a common front door for a lot of the Cloud Pak capabilities, including Event Streams.

The next thing to do is to ask Cloud Pak Foundational services for an ingress, that we can then start configuring with our custom certificate.

oc apply -f 03-management-ingress.yaml -n integration

This will take a couple of minutes.

% oc get operandrequest request-management-ingress
NAME                         AGE   PHASE        CREATED AT
request-management-ingress   58s   Installing   2022-10-26T07:58:49Z

Wait for the management ingress operator to finish installing.

oc wait operandrequest request-management-ingress --for=condition=Ready --timeout=15m
% oc get operandrequest request-management-ingress
NAME                         AGE   PHASE     CREATED AT
request-management-ingress   50s   Running   2022-10-26T07:58:49Z

In the background, a variety of work will then start happening under the covers to set up the auth support. You should wait for this to complete. It can take ten minutes or so.

Wait for the pods in the Common Services namespace to finish starting.

oc get pods -n ibm-common-services

And wait for the setup jobs to complete.

% oc get job -n ibm-common-services
NAME                       COMPLETIONS   DURATION   AGE
iam-onboarding             0/1           9s         9s
oidc-client-registration   0/1           7s         7s
security-onboarding        0/1           10s        10s

% oc get job -n ibm-common-services
NAME                       COMPLETIONS   DURATION   AGE
oidc-client-registration   1/1           9m25s      9m25s
iam-onboarding             1/1           10m        10m
security-onboarding        1/1           10m        10m

Step 5
Configure the ingress to use your certificate

The default management ingress will have created, and (using certmanager) will be managing, a route-cert certificate.

You can find it in the Common Services namespace.

% oc get route-cert -n ibm-common-services
NAME         READY   SECRET             AGE   EXPIRATION
route-cert   True    route-tls-secret   12m   2023-10-26T08:05:47Z

% oc get secret route-tls-secret -n ibm-common-services
NAME               TYPE                DATA   AGE
route-tls-secret   3      6m31s

If you try and replace it, the Operator will immediately revert your changes, so the first step is to edit the management ingress to tell it to stop managing the certificate.

oc patch managementingress default \
    --type merge \
    --patch '{"spec":{"ignoreRouteCert":true}}' \
    -n ibm-common-services

Now that the Operator is ignoring the certificate, you’re free to delete it and replace it with your own custom certificate.

Delete the certificate and secret.

oc delete route-cert -n ibm-common-services
oc delete secret route-tls-secret -n ibm-common-services

Replace the secret with a new one using your certificate.

oc create secret generic route-tls-secret -n ibm-common-services \
    --from-file=ca.crt --from-file=tls.crt --from-file=tls.key

There are a couple of things that don’t automatically notice that the certificate has changed, so you need to prompt them to do something.

You should trigger some certificate secrets to be recreated by deleting them.

oc delete secret ibmcloud-cluster-ca-cert  -n ibm-common-services
oc delete secret management-ingress-ibmcloud-cluster-ca-cert  -n integration

And you should restart the auth pods so that they start using the new certificate.

oc delete pod -l app=auth-idp  -n ibm-common-services

They take a little while to restart, so wait for this to complete.

oc wait --for condition=ready --timeout=900s pod -l app=auth-idp -n ibm-common-services

Step 6
Create Cloud Pak for Integration resources using your certificate

Platform Navigator

Create a new secret using your certificate files that the CP4I components can access.

oc create secret generic my-custom-cert -n integration \
    --from-file=ca.crt --from-file=tls.crt --from-file=tls.key

Create the common Platform Navigator UI.

oc apply -f 04-platform-navigator.yaml -n integration

Notice that it has been configured to use the custom certificate by including this:

    secretName: my-custom-cert

This takes a while to install for the first time, so you should wait for it to be ready.

oc wait PlatformNavigator navigator --for=condition=Ready --timeout=1h -n integration

Event Streams

Create a new secret with the certificate that includes the full chain, suitable for use by the Kafka listeners.

oc create secret tls my-kafka-listeners-cert \
    --cert=full-chain.crt \
    --key=tls.key \
    --dry-run=client -o json | \
    oc apply -n integration -f -

Install the Event Streams operator

oc apply -f 05-operator-event-streams.yaml

Create your Event Streams instance

cpconsoleroute="$(oc get route -n ibm-common-services cp-console -o jsonpath='{}')" \
    envsubst < 06-event-streams.yaml | \
    oc apply -n integration -f -

Notice that the Kafka listeners are configured to use the full chain certificate, by including:

        - tls: true
          type: route
              certificate: tls.crt
              key: tls.key
              secretName: my-kafka-listeners-cert
            type: scram-sha-512
          name: external
          port: 9094

And notice that it has been configured to use the custom single certificate for each of the different REST interfaces that Event Streams provides, by including:

      - name: restproducer
        containerPort: 9080
          certificate: tls.crt
          key: tls.key
          secretName: my-custom-cert
      - name: apicurio
        containerPort: 9080
          certificate: tls.crt
          key: tls.key
          secretName: my-custom-cert
      - name: adminapi
        containerPort: 9080
          certificate: tls.crt
          key: tls.key
          secretName: my-custom-cert

The admin API also needs some additional configuration to tell it to use the external route (which is where the custom certificate is applied, using an OpenShift route set to reencrypt) instead of the default internal service.

      - name: IAM_SERVER_URL
        value: https://${cpconsoleroute}:443

The value for this comes from the foundational services route:

oc get route -n ibm-common-services cp-console -o jsonpath='{}'

Unfortunately, the Event Streams custom resource doesn’t support the normal Kubernetes valueFrom env, which is why I resort to envsubst for this.

You could always edit the yaml file yourself if that is simpler – just remember to:
– add the https:// bit at the start
– include the explicit port number 443 at the end

Step 7

If it’s all worked, you should see the problems described above have gone.

Verify the Kafka listeners

Try creating a topic and credentials you can use with a Kafka application:

oc apply -f 07-kafka-topic.yaml -n integration
oc apply -f 08-kafka-credentials.yaml -n integration

You can then run your Kafka applications. \
    --bootstrap-server $(oc get eventstreams -n integration my-kafka-cluster -o jsonpath='{.status.kafkaListeners[0].bootstrapServers}') \
    --topic TEST \
    --producer-property " required username=testuser password=$(oc get secret testuser -nintegration -ojsonpath={.data.password} | base64 -d);" \
    --producer-property 'sasl.mechanism=SCRAM-SHA-512' \
    --producer-property 'security.protocol=SASL_SSL'

No custom truststore or other system configuration changes needed!

Verify the web interfaces

Try visiting the web console:

oc get eventstreams -n integration my-kafka-cluster -o jsonpath='{.status.endpoints[?("ui")].uri}'
echo "username : `oc get secret -n ibm-common-services platform-auth-idp-credentials -o jsonpath='{.data.admin_username}' | base64 -d`"
echo "password : `oc get secret -n ibm-common-services platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d`\n"

No browser warnings!


With a little additional configuration, you can set up your Event Streams cluster using a trusted certificate that was automatically created for you. It means you can avoid the annoying workarounds that I often see developers putting up with in their development clusters.

For more information, check:

Thanks to Tim Mitchell and Piers Walter for help with this post.

Tags: , , , ,

Comments are closed.