(nearly) 18 years in IBM

July 12th, 2021

I started working at IBM on 6th August 2003. I’m feeling nostalgic as my eighteenth anniversary approaches, so wanted to write about what I’ve been doing all this time.

I’ve been a back-end developer, a support engineer, a tester, a consultant, a (terrible) front-end developer, and much more.

I’ve worked on proprietary software, and I’ve worked on open-source software.

I’ve worked in a large open plan floor, I’ve worked in cubicle bays with half-a-dozen people, and I’ve had my own office. 

I’ve had roles that were fully based at Hursley. I’ve worked from other IBM offices in the UK. I’ve been based at customer sites for months. I’ve had overseas assignments. I’ve had roles that meant travelling to somewhere different every month.

I’ve worked in teams so small they all fit around my dining table for dinner. I’ve worked in teams so large that we needed several coaches for the team social trip to London.

I’ve worked in distributed teams with team members around the world in four different time zones. I’ve worked in teams where we were all in the same office together.

I’ve worked on software that was first released in the 1990s, and I’ve worked on the first releases of brand new products.

The point I’m making… it hasn’t felt like the same job for eighteen years.

Read the rest of this entry »

Tags:
Posted in ibm | No Comments »

Visualizing TensorFlow image classifier behaviour

July 10th, 2021

How to use Scratch to create a visualization that explains what parts of an image a TensorFlow image classifier finds the most significant.

An image classifier recognizes this image as an image of The Doctor.


prediction: The Doctor
confidence: 99.97%

Why? What parts of the image did the classifier recognize as indicating that this is the Doctor?

How could we tell?

Read the rest of this entry »

Tags: , ,
Posted in code | 1 Comment »

Describing Kafka security in AsyncAPI

June 29th, 2021

The new version of AsyncAPI, 2.1.0, was released today. One of the updates is that it lets you describe Kafka security mechanisms and protocols. In this post, I’ll show how you can do this, and how it relates to configuring a Kafka client.

Kafka config means AsyncAPI
sasl.mechanism security.protocol encryption? auth? server protocol security scheme type
unset PLAINTEXT no no kafka
PLAIN SASL_PLAINTEXT no yes, using SASL/PLAIN kafka plain
SCRAM-SHA-256 SASL_PLAINTEXT no yes, using SASL/SCRAM kafka scramSha256
SCRAM-SHA-512 SASL_PLAINTEXT no yes, using SASL/SCRAM kafka scramSha512
OAUTHBEARER SASL_PLAINTEXT no yes, using OAuth kafka oauth2
GSSAPI SASL_PLAINTEXT no yes, using GSSAPI kafka gssapi
unset SSL yes no kafka-secure
PLAIN SASL_SSL yes yes, using SASL/PLAIN kafka-secure plain
SCRAM-SHA-256 SASL_SSL yes yes, using SASL/SCRAM kafka-secure scramSha256
SCRAM-SHA-512 SASL_SSL yes yes, using SASL/SCRAM kafka-secure scramSha512
OAUTHBEARER SASL_SSL yes yes, using OAuth kafka-secure oauth2
GSSAPI SASL_SSL yes yes, using GSSAPI kafka-secure gssapi
unset SSL yes yes, using mutual TLS kafka-secure X509

Read the rest of this entry »

Tags: , ,
Posted in code | No Comments »

How to avoid SSL handshake errors in your Kafka client because of a self-signed cluster CA

June 27th, 2021

You’re trying to connect a Kafka client to a development Apache Kafka cluster which has been quickly set up using a self-signed CA certificate. You don’t have a copy of that CA certificate, and (because it’s not signed by a well-known CA) your Kafka client is failing because of SSL handshake errors.

The error contains messages like
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
and
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

For example:

$ ./bin/kafka-console-consumer.sh \
 --bootstrap-server dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \
 --topic DALE.TOPIC \
 --group dalegrp \
 --consumer-property 'security.protocol=SASL_SSL' \
 --consumer-property 'sasl.mechanism=SCRAM-SHA-512' \
 --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="dale-user" password="pSRtfwTMKNlz";'

[2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 (dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com/9.46.199.58:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:770)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:244)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
	at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:444)
	at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
	at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
	at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
	at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317)
	... 29 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 35 more
Processed a total of 0 messages

I’m assuming that this is just for development purposes, that you know it’s safe to trust the certificate that the Kafka cluster is presenting, and that you’d rather just workaround the error than ask the owner of the Kafka cluster for a copy of their CA.

Read the rest of this entry »

Tags: ,
Posted in code | No Comments »

Event Endpoint Management

June 27th, 2021

Last week, we released the latest version of Event Endpoint Management in IBM Cloud Pak for Integration 2021.2.1. It allows organisations to share and manage access to their Kafka topics. In this post, I want to share a run-through of how it all works.

I’ll start with a high level summary overview, then a walkthrough demo video, and finally share some links to related reading if you’d like more detail.

Overview


click for a larger version of the diagram – numbers in the diagram are described below

Kafka topic owner

This is someone who has a Kafka topic, and is running an application or system that is producing a stream of events to that topic.

They think this stream of events might be useful to other developers in their organisation, so they describe it (using AsyncAPI) and publish this to a catalog where it can be discovered and managed.

  1. creates a Kafka topic and an application that produces events to it
  2. describes and documents their Kafka topic, and the events that are being produced to it
  3. publishes the description of their Kafka topic
  4. pushes the Kafka cluster security info to the Event Gateway service so it can manage access to the topic for the topic owner

App developer

This is someone who is building an application that could benefit from a stream of events.

They are able to discover the event sources that have been shared in their organisation, and get access to them through a self-service Developer Portal.

  1. creates credentials for use in their application
  2. registers new application credentials
  3. updates the Event Gateway service with the new application credentials
  4. creates or configures an application with guidance from the Portal
  5. application connects to the Event Gateway service
  6. application connection routed securely to the Kafka brokers

Read the rest of this entry »

Tags: , ,
Posted in ibm | No Comments »

Win a copy of my “Machine Learning for Kids” book

April 24th, 2021

I’m running a competition to win a copy of my book, “Machine Learning for Kids”.

I mentioned a few months ago that I’ve written a book: “Machine Learning for Kids“.

I’ve got some spare copies of it that need a good home, so I thought it might be fun to run a competition!

I’ve got five copies that I’m going to give away in this competition.



To enter, I’m looking for new ideas for teaching children about AI and machine learning.

This could be an idea for a new machine learning project worksheet. You can see machinelearningforkids.co.uk/worksheets for examples of the sorts of thing this could cover. You could contribute a new worksheet, or if you’d prefer, you can just explain your idea for a new project worksheet and what students would learn from it.

This can include an idea for a new feature or capability on the Machine Learning for Kids website. You could contribute a design for the new capability, or you can just explain how it would work and what students would learn from it.

To take part, email your ideas to competition@machinelearningforkids.co.uk by 4th June 2021.

I’ll choose my five favourite ideas, and post a free paperback copy of my book to each of the five winners.

Full details and terms below, but please note the really big one: UK residents only, please. Sorry, but I don’t want to get into international shipping – so please only enter if you’ve got a UK address I could post a book to!

Read the rest of this entry »

Posted in misc | 1 Comment »

Building a Question Answering game in Scratch

April 17th, 2021

I added a new project worksheet to Machine Learning for Kids today.

It has step-by-step instructions for how to make a quiz show game in Scratch that uses a machine learning model to understand questions on any topic the student chooses, and find the answer in Wikipedia pages.

It’s a fun little project, super simple to make, and works surprisingly well. It doesn’t get every question right, but it does a lot better than I expected.

I don’t normally write blog posts when I write new ML for Kids worksheets, but this one was a bit interesting.

Read the rest of this entry »

Tags: , ,
Posted in tech | No Comments »

Machine learning workshop for school teachers

April 2nd, 2021

This week I ran a remote workshop for school teachers about machine learning and artificial intelligence. It was organised with University College London as part of a series of activities they are running to celebrate the CS Expo: 40+ years of UCL Computer Science.

It was quite a long session, as we wanted it to be a hands-on practical CPD (Continuing Professional Development) workshop rather than just me giving a short talk. In the 90-minute workshop, we made two separate AI projects, which was a chance to see and contrast a few different approaches.

Read the rest of this entry »

Tags: ,
Posted in misc | No Comments »

« Older Entries