A quick how-to for setting up Event Streams with trusted certificates when running a development project.
Problem
You’re working on a project using IBM Event Streams. It’s just a development project, so you’re not using an SSL certificate signed by your real, trusted, corporate signer.
Everything works, but…
You get errors like these every time you access the web tooling – which you have to click through.
And you get errors like these from your Kafka client applications – which you have to configure with a custom truststore to avoid (although, if you do need to do that, I have a guide to help!)
[2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 (dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com/9.46.199.58:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) at java.base/java.security.AccessController.doPrivileged(AccessController.java:770) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
Explanation
The reason for these errors is because by default, Cloud Pak for Integration – in the absence of a certificate you’d rather use – has generated new self-signed certificates to use.
Your browser and client applications don’t trust this new self-signed certificate by default.
Solution
TLDR: Use a free certificate from Let’s Encrypt. When you set up Event Streams, configure it to use that certificate. That will (almost certainly) be trusted by all of your apps, and you won’t have to go through the annoying workarounds.
The rest of this post is just a guided walkthrough explaining how to do that.
To keep things simple, I’m going to assume that you’re starting from scratch – you have a brand new, vanilla, OpenShift cluster.
- 1 – Get a certificate
- 2 – Install the CP4I operators
- 3 – Prepare the namespace
- 4 – Start setting up Foundational Services
- 5 – Set up ingress using your certificate
- 6 – Create CP4I components
- 7 – Verify
Note:
Some of the commands in this post use files that you can find at
github.com/dalelane/eventstreams-valid-ssl.
Commands you should run will be highlighted like this
Commands to see what is happening will be shown like this
Step 1
Get a certificate
ROKS
If you are using a managed OpenShift cluster in IBM Cloud (“ROKS”), a certificate is automatically created for you when you create the cluster. It’s already available as a Secret in the openshift-ingress namespace, and you can use that.
SECRET_NAME=$(oc get secret -n openshift-ingress -o=jsonpath='{.items[?(@.metadata.annotations.ingress\.cloud\.ibm\.com/cert-source=="ibm")].metadata.name}')
echo "Let's Encrypt certificate is available in the $SECRET_NAME secret in the openshift-ingress namespace"
You can download this to a few files on your computer.
First, the key:
oc get secret -n openshift-ingress $SECRET_NAME -o=jsonpath="{.data['tls\.key']}" | base64 -d > tls.key
Then the full chain, which you’ll need to split into the certificate and the CA’s.
oc get secret -n openshift-ingress $SECRET_NAME -o=jsonpath="{.data['tls\.crt']}" | base64 -d > full-chain.crt
split -p "-----BEGIN CERTIFICATE-----" full-chain.crt cert-
mv cert-aa tls.crt
cat cert-ab cert-ac > ca.crt
rm cert-ab cert-ac
You should now have three files:
– tls.key
– tls.crt
– ca.crt
Other
If you’re running OpenShift somewhere else, you will need to create the certificate yourself. See the documentation for more guidance.
Step 2
Install the Cloud Pak for Integration operators
Add the IBM software catalog to your OpenShift cluster.
oc apply -f 01-ibm-catalog-source.yaml
Wait for the operator catalog to be updated with the IBM operators, then install the Cloud Pak for Integration operator.
oc apply -f 02-operator-platform-navigator.yaml
Wait for the IBM Cloud Pak for Integration operator and it’s pre-reqs (IBM Cloud Pak foundational services, and IBM NamespaceScope Operator) finish installing.
Step 3
Prepare the namespace
I’m going to use a namespace called integration in this walkthrough. You can obviously change this to anything you’d like in all the subsequent commands.
Create the namespace.
oc new-project integration
Create an entitled registry key in the namespace, that you’ll need to deploy IBM software.
oc create secret docker-registry ibm-entitlement-key \
--docker-username=cp \
--docker-password=$IBM_ENTITLEMENT_KEY \
--docker-server=cp.icr.io \
--namespace=integration --dry-run=client -o yaml | oc apply -f -
Step 4
Start creating Cloud Pak Foundational Services
Cloud Pak Foundational Services provides a common front door for a lot of the Cloud Pak capabilities, including Event Streams.
The next thing to do is to ask Cloud Pak Foundational services for an ingress, that we can then start configuring with our custom certificate.
oc apply -f 03-management-ingress.yaml -n integration
This will take a couple of minutes.
% oc get operandrequest request-management-ingress NAME AGE PHASE CREATED AT request-management-ingress 58s Installing 2022-10-26T07:58:49Z
Wait for the management ingress operator to finish installing.
oc wait operandrequest request-management-ingress --for=condition=Ready --timeout=15m
% oc get operandrequest request-management-ingress NAME AGE PHASE CREATED AT request-management-ingress 50s Running 2022-10-26T07:58:49Z
In the background, a variety of work will then start happening under the covers to set up the auth support. You should wait for this to complete. It can take ten minutes or so.
Wait for the pods in the Common Services namespace to finish starting.
oc get pods -n ibm-common-services
And wait for the setup jobs to complete.
% oc get job -n ibm-common-services NAME COMPLETIONS DURATION AGE iam-onboarding 0/1 9s 9s oidc-client-registration 0/1 7s 7s security-onboarding 0/1 10s 10s % oc get job -n ibm-common-services NAME COMPLETIONS DURATION AGE oidc-client-registration 1/1 9m25s 9m25s iam-onboarding 1/1 10m 10m security-onboarding 1/1 10m 10m
Step 5
Configure the ingress to use your certificate
The default management ingress will have created, and (using certmanager) will be managing, a route-cert certificate.
You can find it in the Common Services namespace.
% oc get certificates.certmanager.k8s.io route-cert -n ibm-common-services NAME READY SECRET AGE EXPIRATION route-cert True route-tls-secret 12m 2023-10-26T08:05:47Z % oc get secret route-tls-secret -n ibm-common-services NAME TYPE DATA AGE route-tls-secret kubernetes.io/tls 3 6m31s
If you try and replace it, the Operator will immediately revert your changes, so the first step is to edit the management ingress to tell it to stop managing the certificate.
oc patch managementingress default \
--type merge \
--patch '{"spec":{"ignoreRouteCert":true}}' \
-n ibm-common-services
Now that the Operator is ignoring the certificate, you’re free to delete it and replace it with your own custom certificate.
Delete the certificate and secret.
oc delete certificates.certmanager.k8s.io route-cert -n ibm-common-services oc delete secret route-tls-secret -n ibm-common-services
Replace the secret with a new one using your certificate.
oc create secret generic route-tls-secret -n ibm-common-services \
--from-file=ca.crt --from-file=tls.crt --from-file=tls.key
There are a couple of things that don’t automatically notice that the certificate has changed, so you need to prompt them to do something.
You should trigger some certificate secrets to be recreated by deleting them.
oc delete secret ibmcloud-cluster-ca-cert -n ibm-common-services oc delete secret management-ingress-ibmcloud-cluster-ca-cert -n integration
And you should restart the auth pods so that they start using the new certificate.
oc delete pod -l app=auth-idp -n ibm-common-services
They take a little while to restart, so wait for this to complete.
oc wait --for condition=ready --timeout=900s pod -l app=auth-idp -n ibm-common-services
Step 6
Create Cloud Pak for Integration resources using your certificate
Platform Navigator
Create a new secret using your certificate files that the CP4I components can access.
oc create secret generic my-custom-cert -n integration \
--from-file=ca.crt --from-file=tls.crt --from-file=tls.key
Create the common Platform Navigator UI.
oc apply -f 04-platform-navigator.yaml -n integration
Notice that it has been configured to use the custom certificate by including this:
spec:
tls:
secretName: my-custom-cert
This takes a while to install for the first time, so you should wait for it to be ready.
oc wait PlatformNavigator navigator --for=condition=Ready --timeout=1h -n integration
Event Streams
Create a new secret with the certificate that includes the full chain, suitable for use by the Kafka listeners.
oc create secret tls my-kafka-listeners-cert \
--cert=full-chain.crt \
--key=tls.key \
--dry-run=client -o json | \
oc apply -n integration -f -
Install the Event Streams operator
oc apply -f 05-operator-event-streams.yaml
Create your Event Streams instance
cpconsoleroute="$(oc get route -n ibm-common-services cp-console -o jsonpath='{.spec.host}')" \
envsubst < 06-event-streams.yaml | \
oc apply -n integration -f -
Notice that the Kafka listeners are configured to use the full chain certificate, by including:
spec:
strimziOverrides:
kafka:
listeners:
- tls: true
type: route
configuration:
brokerCertChainAndKey:
certificate: tls.crt
key: tls.key
secretName: my-kafka-listeners-cert
authentication:
type: scram-sha-512
name: external
port: 9094
And notice that it has been configured to use the custom single certificate for each of the different REST interfaces that Event Streams provides, by including:
spec:
restProducer:
endpoints:
- name: restproducer
containerPort: 9080
certOverrides:
certificate: tls.crt
key: tls.key
secretName: my-custom-cert
apicurioRegistry:
endpoints:
- name: apicurio
containerPort: 9080
certOverrides:
certificate: tls.crt
key: tls.key
secretName: my-custom-cert
adminApi:
endpoints:
- name: adminapi
containerPort: 9080
certOverrides:
certificate: tls.crt
key: tls.key
secretName: my-custom-cert
The admin API also needs some additional configuration to tell it to use the external route (which is where the custom certificate is applied, using an OpenShift route set to reencrypt) instead of the default internal service.
spec:
adminApi:
env:
- name: IAM_SERVER_URL
value: https://${cpconsoleroute}:443
The value for this comes from the foundational services route:
oc get route -n ibm-common-services cp-console -o jsonpath='{.spec.host}'
Unfortunately, the Event Streams custom resource doesn’t support the normal Kubernetes valueFrom env, which is why I resort to envsubst for this.
You could always edit the yaml file yourself if that is simpler – just remember to:
– add the https:// bit at the start
– include the explicit port number 443 at the end
Step 7
Verify
If it’s all worked, you should see the problems described above have gone.
Verify the Kafka listeners
Try creating a topic and credentials you can use with a Kafka application:
oc apply -f 07-kafka-topic.yaml -n integration oc apply -f 08-kafka-credentials.yaml -n integration
You can then run your Kafka applications.
kafka-console-producer.sh \
--bootstrap-server $(oc get eventstreams -n integration my-kafka-cluster -o jsonpath='{.status.kafkaListeners[0].bootstrapServers}') \
--topic TEST \
--producer-property "sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=testuser password=$(oc get secret testuser -nintegration -ojsonpath={.data.password} | base64 -d);" \
--producer-property 'sasl.mechanism=SCRAM-SHA-512' \
--producer-property 'security.protocol=SASL_SSL'
No custom truststore or other system configuration changes needed!
Verify the web interfaces
Try visiting the web console:
oc get eventstreams -n integration my-kafka-cluster -o jsonpath='{.status.endpoints[?(@.name=="ui")].uri}'
echo "username : `oc get secret -n ibm-common-services platform-auth-idp-credentials -o jsonpath='{.data.admin_username}' | base64 -d`"
echo "password : `oc get secret -n ibm-common-services platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d`\n"
No browser warnings!
Summary
With a little additional configuration, you can set up your Event Streams cluster using a trusted certificate that was automatically created for you. It means you can avoid the annoying workarounds that I often see developers putting up with in their development clusters.
For more information, check:
Thanks to Tim Mitchell and Piers Walter for help with this post.
Tags: apachekafka, eventstreams, ibmeventstreams, kafka, ssl