How to run a tiny local Kafka cluster using IBM Event Streams images
For local development on Kafka projects, I always run the public open source builds of ZooKeeper and Kafka as Java processes directly on my laptop (similar to steps described in the Apache Kafka Quickstart).
But for a project this week, I needed to verify something with the distribution of Kafka that comes with IBM Event Streams.
I used a simple Docker Compose setup for this. I’ll use this post to share how I did it.
A quick tip for how to give a developer access to the IBM Event Streams UI only for the Kafka topics used by their application, and not everything else.
Imagine I’m a Kafka cluster admin. I’m running a cluster with a variety of topics on it.
One of my developers is responsible for the flight tracking app, and wants to use the Event Streams UI. But I don’t want them to be able to access the other sensitive topics for other applications.
I can create them their own login for the UI, that only lets them see their own topics.
The permissions I want to give them are:
- operation: Read
resource:
name: FLIGHT.
patternType: prefix
type: topic
(Remember, managing my Kafka cluster through Kubernetes resources is a good fit with a CI/CD workflow.)
In this post, I share a step-by-step guide for how to use IBM DataStage to merge JSON messages from multiple different Apache Kafka topics, into a single joined-up stream of events.
A quick how-to for setting up Event Streams with trusted certificates when running a development project.
You’re working on a project using IBM Event Streams. It’s just a development project, so you’re not using an SSL certificate signed by your real, trusted, corporate signer.
Everything works, but…
You get errors like these every time you access the web tooling – which you have to click through.
And you get errors like these from your Kafka client applications – which you have to configure with a custom truststore to avoid (although, if you do need to do that, I have a guide to help!)
[2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 (dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com/9.46.199.58:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) at java.base/java.security.AccessController.doPrivileged(AccessController.java:770) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
In this post, I’ll share how you can get your own Event Endpoint Management demo instance with just seven minutes of work.
(Seven minutes of hands-on-keyboard time… there is a lot of waiting-for-stuff-to-run time, but it doesn’t sound so impressive if I include waiting time!)
We’ve been running a virtual event this week to explain the capabilities of IBM’s Cloud Pak for Integration.
One of these is Event Streams, so I gave an overview of the Event Streams Operator.
But what it really reminded me is that I miss going to conferences and tech events. I don’t want to sound ungrateful for what I’m sure has been a huge amount of work for event organisers in the pivot to online events. It’s great that we can still do events at all, and that organisers are still trying out ways to make it interactive, to enable panels and Q&A sessions.
Operators bring a lot of benefits as a way of managing complex software systems in a Kubernetes cluster. In this post, I want to illustrate one in particular: the way that custom resources (and declarative approaches to managing systems in general) enable easy integration with source control and a CI/CD pipeline.
I’ll be using IBM Event Streams as my example here, but the same principles will be true for many Kubernetes Operators, in particular, the open-source Strimzi Kafka Operator that Event Streams is based on.
Installing operators in Red Hat OpenShift from the CLI is much easier with the new kubectl-operator plugin. Here’s an example of how you can use it to install the Event Streams Operator.
Installing operators in OpenShift from the CLI is a little fiddly. It’s possible, but you have to create a bunch of custom resources that aren’t entirely intuitive, like Subscriptions and OperatorGroups.
It’s easy if you’re using the OpenShift Console web UI, as it does this all for you so you don’t need to worry about it. But sometimes you want to do things from the command line. And the new kubectl-operator plugin looks like it’ll make that much simpler.
I had a quick play with it this evening, and it let me get the Event Streams operator running with three commands. (Compare this with the OpenShift Console web UI equivalent in my Event Streams demo video).
