A Kafka Developer’s Guide to AsyncAPI

March 30th, 2021

How Kafka developers can use the AsyncAPI specification to describe how their applications are using Kafka topics.

In my post “Why should you document your Kafka topics?” last week, I wrote about the benefits of documenting your Kafka event sources, and mentioned a few of the problems that this can help with.

In this post, I want to show you how you can document the API for your Kafka event sources by creating AsyncAPI documents.

You don’t necessarily have to learn the AsyncAPI specification – tools such as the new Event Endpoint Management capability that I work on in Cloud Pak for Integration make it easy to document APIs with user-friendly forms that generate AsyncAPI documents for you. However, some developers will want to know more about what is happening under the covers, so here is an introduction.


youtu.be/Ni5tCY9r0TY

Read the rest of this entry »

Migrating your Apache Kafka cluster using MirrorMaker 2

March 24th, 2021

You have a Kafka cluster that you have been using for a while. Your cluster has many topics, and the topics have many messages.

Now you’ve decided to move and start using a new, different Kafka cluster somewhere else.

How can you take your topics with you?

Huge thanks to Andrew Borley for co-writing this with me. Useful insights in here probably came from him, the mistakes from me.

Read the rest of this entry »

AsyncAPI plugin for Node-RED

March 21st, 2021

screenshot

I’ve been tinkering with a new AsyncAPI plugin for Node-RED as a side project over the last couple of weeks. Time to share what I’ve got working so far.

Node-RED is an open-source visual programming tool. You assemble flows on a canvas from a palette of nodes, that you customize and then wire together. That makes it ideal for quick prototyping.

There are nodes for different types of servers and devices, which makes it great for quick integration projects. This includes nodes for sending and receiving messages using protocols like Kafka and MQTT.

Normally, this means having to choosing a node from the palette, dragging it onto the canvas, clicking into it, and customizing it: filling in the connection details for the broker and topic you want to use.

The idea of this plugin is to do all of that for you, if you have an AsyncAPI specification for your topics.

The plugin can generate and customize the nodes for you, based on your spec – making rapid prototyping based on the spec even quicker.

Read the rest of this entry »

I wrote a book

January 24th, 2021

It’s called “Machine Learning for Kids: A Project-Based Introduction to Artificial Intelligence”.

It’s a hands-on, application-based introduction to machine learning and artificial intelligence that guides young readers through creating compelling AI-powered games and applications using the Scratch programming language.

Since starting the Machine Learning for Kids site, I’ve written project worksheets to inspire students and teachers what can be built using the tool. By making them freely available as Creative Commons-licensed MS Word docs, they’ve been a jumping off point to help teachers and code-club leaders to create their own lessons and activities.

As I’ve written the worksheets with schools and code clubs in mind, that introduced constraints.

Each worksheet is self-contained – many schools will only have time in their timetable/curriculum for one, or maybe two, AI projects, so a lot of the projects retread some of the same basics. None of them build on, or even refer to, any of the other worksheets. They also need to be short activities, so that they can be completed within a school lesson.

Writing a book version of Machine Learning for Kids was a chance to do something for a different audience: this time aimed at a child at home with their parents.

This means I didn’t have the same constraints as the worksheets on the site. It’s still based on explaining machine learning in a hands-on way through making projects in Scratch. But there’s a flow between the projects in the book. They’re in an intentional order, and there is a continuation between them. Each project builds upon the projects that came before it.

Some of the projects take a bit longer as they don’t need to be done in one sitting. I have more time and space to explain the ideas and to give the real-world context for each project. As each project doesn’t need to work as an introduction, it means the later chapters can get into more advanced topics that none of the project worksheets on the site go near, like accuracy, recall, and confidence matrices.

It’s been a lot of work. A lot more than I expected. Over two years of work. And not just by me: I had no idea how many people would be involved in making the book into a real thing. I’ve not really worked with editors before, and it has been a fascinating experience. They made my rambling gibbering so so much better that I’m almost embarrassed that only my name is on the cover. There’s no way the finished thing would be nearly as good without their work.

There were a few points where I wondered if it’d ever actually see the light of day – but it’s finally available. (Well, the e-book is available now, but the printed version is still a couple of weeks away).

I hope people find it useful! I am proud of it. I’m particularly proud of the Foreword, which I didn’t even write. It was very generously written by Grady Booch, and it’s the perfect inspirational start to what I wanted the book to be.

It’s very strange to see something I’ve written in online bookshops. It’s in Amazon, Waterstones, and WHSmith. That feels a bit weird. I hope that at some point I’ll get to see a printed copy in a real bookshop, but I suspect that won’t be any time soon!

Looking back at Machine Learning for Kids in 2020

January 2nd, 2021

A review of what I did on Machine Learning for Kids in 2020.

Happy New Year!

At this time of year, it’s traditional to get a bit reflective, so I thought I’d look over the work I did on ML for Kids in 2020.
Read the rest of this entry »

Introducing ‘Machine Learning for Kids’ to teachers

December 17th, 2020

I gave a short talk about Machine Learning for Kids last week as part of an online conference run by Somerset eLIM. Here’s the recording.


youtu.be/8St1REZbE5w

I started with a couple of definitions, then demonstrated a variety of projects that I’ve seen primary school students make, and finally walked people through a hands-on demo so they could try it out for themselves.

Developer Guide: Machine Learning for Kids

November 27th, 2020

A run-through of the DEVELOPMENT.md guide.

In this video, I go from zero to a running Machine Learning for Kids website (including installing all the necessary dependencies and building the site from source).


youtu.be/Ss3e6yCWOhU

Describing Kafka with AsyncAPI

November 27th, 2020

In this post, I want to describe how to use AsyncAPI to document how you’re using Apache Kafka. There are already great AsyncAPI “Getting Started” guides, but it supports a variety of protocols, and I haven’t found an introduction written specifically from the perspective of a Kafka user.

I’ll start with a description of what AsyncAPI is.

“an open source initiative … goal is to make working with Event-Driven Architectures as easy as it is to work with REST APIs … from documentation to code generation, from discovery to event management”

asyncapi.com/docs

The most obvious initial aspect is that it is a way to document how you’re using Kafka topics, but the impact is broader than that: a consistent approach to documentation enables an ecosystem that includes things like automated code generation and discovery.

Read the rest of this entry »