I was lucky enough to be invited onto a couple of podcasts this month to talk about Machine Learning for Kids.
Read the rest of this entry »
Talking about machine learning for kids
September 18th, 2021Stock price events for Kafka
August 28th, 2021I’ve made a Kafka Connect source connector for sending “real-time” (not really) stock price events to a Kafka topic.
Choose your stock (e.g. “IBM“) and run the Connector pointed at your topic, and every minute a JSON event will be produced like:
{"open":137.9,"high":137.9,"low":137.9,"close":137.9,"volume":500,"timestamp":1629421200,"datetime":"2021-08-19 20:00:00"}
(nearly) 18 years in IBM
July 12th, 2021I started working at IBM on 6th August 2003. I’m feeling nostalgic as my eighteenth anniversary approaches, so wanted to write about what I’ve been doing all this time.
I’ve been a back-end developer, a support engineer, a tester, a consultant, a (terrible) front-end developer, and much more.
I’ve worked on proprietary software, and I’ve worked on open-source software.
I’ve worked in a large open plan floor, I’ve worked in cubicle bays with half-a-dozen people, and I’ve had my own office.
I’ve had roles that were fully based at Hursley. I’ve worked from other IBM offices in the UK. I’ve been based at customer sites for months. I’ve had overseas assignments. I’ve had roles that meant travelling to somewhere different every month.
I’ve worked in teams so small they all fit around my dining table for dinner. I’ve worked in teams so large that we needed several coaches for the team social trip to London.
I’ve worked in distributed teams with team members around the world in four different time zones. I’ve worked in teams where we were all in the same office together.
I’ve worked on software that was first released in the 1990s, and I’ve worked on the first releases of brand new products.
The point I’m making… it hasn’t felt like the same job for eighteen years.
Visualizing TensorFlow image classifier behaviour
July 10th, 2021How to use Scratch to create a visualization that explains what parts of an image a TensorFlow image classifier finds the most significant.
An image classifier recognizes this image as an image of The Doctor.
prediction: The Doctor
confidence: 99.97%
Why? What parts of the image did the classifier recognize as indicating that this is the Doctor?
How could we tell?
Describing Kafka security in AsyncAPI
June 29th, 2021The new version of AsyncAPI, 2.1.0, was released today. One of the updates is that it lets you describe Kafka security mechanisms and protocols. In this post, I’ll show how you can do this, and how it relates to configuring a Kafka client.
| Kafka config | means | AsyncAPI | |||
|---|---|---|---|---|---|
sasl.mechanism |
security.protocol |
encryption? | auth? | server protocol |
security scheme type |
| unset | PLAINTEXT |
no | no | kafka |
|
PLAIN |
SASL_PLAINTEXT |
no | yes, using SASL/PLAIN | kafka |
plain |
SCRAM-SHA-256 |
SASL_PLAINTEXT |
no | yes, using SASL/SCRAM | kafka |
scramSha256 |
SCRAM-SHA-512 |
SASL_PLAINTEXT |
no | yes, using SASL/SCRAM | kafka |
scramSha512 |
OAUTHBEARER |
SASL_PLAINTEXT |
no | yes, using OAuth | kafka |
oauth2 |
GSSAPI |
SASL_PLAINTEXT |
no | yes, using GSSAPI | kafka |
gssapi |
| unset | SSL |
yes | no | kafka-secure |
|
PLAIN |
SASL_SSL |
yes | yes, using SASL/PLAIN | kafka-secure |
plain |
SCRAM-SHA-256 |
SASL_SSL |
yes | yes, using SASL/SCRAM | kafka-secure |
scramSha256 |
SCRAM-SHA-512 |
SASL_SSL |
yes | yes, using SASL/SCRAM | kafka-secure |
scramSha512 |
OAUTHBEARER |
SASL_SSL |
yes | yes, using OAuth | kafka-secure |
oauth2 |
GSSAPI |
SASL_SSL |
yes | yes, using GSSAPI | kafka-secure |
gssapi |
| unset | SSL |
yes | yes, using mutual TLS | kafka-secure |
X509 |
How to avoid SSL handshake errors in your Kafka client because of a self-signed cluster CA
June 27th, 2021You’re trying to connect a Kafka client to a development Apache Kafka cluster which has been quickly set up using a self-signed CA certificate. You don’t have a copy of that CA certificate, and (because it’s not signed by a well-known CA) your Kafka client is failing because of SSL handshake errors.
The error contains messages like
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
and
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
For example:
$ ./bin/kafka-console-consumer.sh \ --bootstrap-server dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \ --topic DALE.TOPIC \ --group dalegrp \ --consumer-property 'security.protocol=SASL_SSL' \ --consumer-property 'sasl.mechanism=SCRAM-SHA-512' \ --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="dale-user" password="pSRtfwTMKNlz";' [2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 (dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com/9.46.199.58:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) at java.base/java.security.AccessController.doPrivileged(AccessController.java:770) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:244) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480) at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:444) at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103) at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77) at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54) at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317) ... 29 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 35 more Processed a total of 0 messages
I’m assuming that this is just for development purposes, that you know it’s safe to trust the certificate that the Kafka cluster is presenting, and that you’d rather just workaround the error than ask the owner of the Kafka cluster for a copy of their CA.
Event Endpoint Management
June 27th, 2021Last week, we released the latest version of Event Endpoint Management in IBM Cloud Pak for Integration 2021.2.1. It allows organisations to share and manage access to their Kafka topics. In this post, I want to share a run-through of how it all works.
I’ll start with a high level summary overview, then a walkthrough demo video, and finally share some links to related reading if you’d like more detail.
Overview
click for a larger version of the diagram – numbers in the diagram are described below
Kafka topic owner
This is someone who has a Kafka topic, and is running an application or system that is producing a stream of events to that topic.
They think this stream of events might be useful to other developers in their organisation, so they describe it (using AsyncAPI) and publish this to a catalog where it can be discovered and managed.
- creates a Kafka topic and an application that produces events to it
- describes and documents their Kafka topic, and the events that are being produced to it
- publishes the description of their Kafka topic
- pushes the Kafka cluster security info to the Event Gateway service so it can manage access to the topic for the topic owner
App developer
This is someone who is building an application that could benefit from a stream of events.
They are able to discover the event sources that have been shared in their organisation, and get access to them through a self-service Developer Portal.
- creates credentials for use in their application
- registers new application credentials
- updates the Event Gateway service with the new application credentials
- creates or configures an application with guidance from the Portal
- application connects to the Event Gateway service
- application connection routed securely to the Kafka brokers
Win a copy of my “Machine Learning for Kids” book
April 24th, 2021I’m running a competition to win a copy of my book, “Machine Learning for Kids”.
I mentioned a few months ago that I’ve written a book: “Machine Learning for Kids“.
I’ve got some spare copies of it that need a good home, so I thought it might be fun to run a competition!
I’ve got five copies that I’m going to give away in this competition.
To enter, I’m looking for new ideas for teaching children about AI and machine learning.
This could be an idea for a new machine learning project worksheet. You can see machinelearningforkids.co.uk/worksheets for examples of the sorts of thing this could cover. You could contribute a new worksheet, or if you’d prefer, you can just explain your idea for a new project worksheet and what students would learn from it.
This can include an idea for a new feature or capability on the Machine Learning for Kids website. You could contribute a design for the new capability, or you can just explain how it would work and what students would learn from it.
To take part, email your ideas to competition@machinelearningforkids.co.uk by 4th June 2021.
I’ll choose my five favourite ideas, and post a free paperback copy of my book to each of the five winners.
Full details and terms below, but please note the really big one: UK residents only, please. Sorry, but I don’t want to get into international shipping – so please only enter if you’ve got a UK address I could post a book to!