Using IBM Event Automation with watsonx

May 29th, 2024

In this post, I want to share examples of how watsonx can enhance the event processing flows you create using IBM Event Processing.

I’ll start by describing how Event Processing and watsonx complement each other.

Then I’ll share a couple of simple examples of what this looks like in action.

Finally, I’ll walkthrough how I built the example flows to show you how you can try doing something like this for yourself, and share tips for how to create flows like this.

Read the rest of this entry »

Using books data in Scratch

May 19th, 2024

In this post, I want to share a Scratch extension that I’ve been working on today: enabling access to books data from the OpenLibrary API through new Scratch blocks.

Most of the work I do on Machine Learning for Kids involves adding machine learning models into Scratch. To enable students to create interesting projects, it also helps to make it easier to get external data into Scratch that they can use for training and classifying. A few examples of where I’ve done this in the past include creating Scratch blocks to access weather data, data from Spotify, and data from Wikipedia.

New blocks

The new blocks I’ve worked on today use the OpenLibrary API to enable access to information about books.

Read the rest of this entry »

Processing XML with Kafka Connect

May 15th, 2024

I spoke at Devoxx UK last week, about how to process XML data using a Kafka Connect pipeline.

This was based on some work I did last year, but it was good to get a chance to share it with a new audience.


youtu.be/NfYHE2i0-es

Processing Apache Avro-serialized messages from Kafka using IBM App Connect Enterprise

May 13th, 2024

IBM App Connect Enterprise (ACE) is a broker for developing and hosting high-throughput, high-scale integrations between a large number of applications and systems, including Apache Kafka.

In this post, I’ll describe how to use App Connect Enterprise to process Kafka messages that were serialized using Apache Avro schemas.

screenshot

This is an update to an earlier version of this post, reflecting updates to the sample code.

Background

Best practice when using Apache Kafka is to define Apache Avro schemas with a definition of the structure of your Kafka messages, and to store those schemas in a central registry that client applications can access at runtime.

If you want to use IBM App Connect Enterprise to develop and host integrations for processing those Kafka messages, you need App Connect to know how to:

  • retrieve the Avro schemas it needs using schema registry REST APIs
  • use the schemas to turn the binary stream of bytes on your Kafka topics into structured objects that ACE can manipulate and process

Read the rest of this entry »

Using IBM Event Automation with Azure Event Hubs

April 26th, 2024

IBM Event Automation helps companies to accelerate their event-driven projects wherever businesses are on their journey. It provides multiple components (Event Streams, Event Endpoint Management, and Event Processing) which together lay the foundation of an event-driven architecture that can unlock the value of the streams of events that businesses have.

A key goal of Event Automation is to be composable. The three components can be used together, or they can each be used to extend and enhance an existing event-driven deployment.

Today, I demonstrated some of the Event Automation components working with Azure Event Hubs for Apache Kafka. As Event Hubs provides a Kafka interface to Azure’s data streaming service, it obviously can be used with Event Automation. But it can be helpful to inspire people by showing it for real, so even demos of obvious things can be valuable.

For example, Event Endpoint Management can enhance the value of topics in Event Hubs by offering management and governance, and by enabling governed reuse of those topics. Event Processing makes it easy to get insights from the events on Event Hubs topics, by providing an intuitive low-code authoring canvas to process them.

If I was going to be running this for a while and wanted to optimise for my applications in Azure, I would likely have set this up like this, with the Event Gateways deployed close to the Azure Kafka endpoints.

Read the rest of this entry »

Hoops (a Machine Learning for Kids worksheet)

April 21st, 2024

Machine Learning for Kids is intended to be an open creative sandbox to let students invent their own AI-powered projects. But in order to enable that, I create more prescriptive project worksheets to inspire and show what is possible.

I’ve just written another worksheet based around regression models – a model type that I added support for in February.

This project is based on shooting basketballs.

Read the rest of this entry »

Using Mirror Maker 2 with IBM Event Streams to migrate to a new cluster

April 18th, 2024

This is the sixth in a series of blog posts sharing examples of ways to use Mirror Maker 2 with IBM Event Streams.

Mirror Maker 2 is a powerful and flexible tool for moving Kafka events between Kafka clusters.

For this sixth post, I’ll look at using Mirror Maker to migrate your Kafka cluster to a new region.

I’ve broken this down into multiple stages. For each stage, I’ll explain the intent and share a demo script I’ve created to let you try this for yourself.

Read the rest of this entry »

Shoot the bug (a Machine Learning for Kids worksheet)

April 17th, 2024

Students are free to devise their own projects in Machine Learning for Kids, but I also write project worksheets to help inspire students and teachers.

This evening, I’ve written a new worksheet based around regression models. (I wrote about adding support for regression models to the site a couple months ago).

The premise for this project is sort of Space Invaders. Except with only a single Space Invader. And it’s a cute little bug.

The aim of the game is to shoot at the bug.

Read the rest of this entry »