Using Node-RED with IBM Event Streams

June 28th, 2019


Click to enlarge

IBM Event Streams is the distributed streaming real-time data platform Apache Kafka, from IBM.

Node-RED is a visual flow-based development tool, with nodes that you drag and drop onto a canvas and wire together. It’s useful for loads of tasks, such as quick and flexible prototyping.

In this post, I’ll show how Event Streams and Node-RED work well together. You can use Node-RED to quickly and easily create flows that consume messages from Kafka topics, or that process events from different sources and produce the output to Kafka topics.

Read the rest of this entry »

Curated sample training datasets for Machine Learning for Kids

June 26th, 2019

Machine Learning for Kids now includes support for a curated collection of training data sets, to enable children to create different types of machine learning projects.


Click to enlarge

The tool lets children make things using machine learning. The principle I’ve worked to is that children train their own machine learning models, as doing this is a great way to teach them about how this tech works.

Preparing their own training data is a useful exercise, but it is time-consuming. Project worksheets I’ve written so far have all been written with the assumption that the student will prepare the training data within a single lesson. This has been a limiting factor on the kinds of ML projects I’ve been able to include.

Read the rest of this entry »

Are indie games better value than AAA games?

June 23rd, 2019

This post started life as a debate with friends about whether big triple-A games are better value than cheaper indie games. We didn’t have data, the debate was just opinions. But it stuck with me, so I decided to collect data to prove I was right. 🙂

The plan was to plot time I spend playing games against how much money I spent on them, and use the clear correlation to prove my point. That didn’t work. I didn’t find much of a pattern, but it’s been a while since I’ve done this sort of quantified-self thing and collecting the data was a pain so I’m sharing it anyway!

To start with, this graph plots the cost of each game (x axis) against the number of hours I’ve spent playing them (y axis).


Cost against Hours played – click for larger version

Read the rest of this entry »

How to increase your social impact

June 16th, 2019

This is a talk I gave at an event about how we are able to make a social impact through volunteering and community projects.

I’ve written before about how I made Machine Learning for Kids. But this talk had a different focus.

For this presentation, I looked to see if there are any general lessons that could be learned from my experience, to let me offer a little advice for people working in large companies like IBM about how to increase the impact of their volunteering efforts.

Read the rest of this entry »

Using kafkacat and kaf with IBM Event Streams

June 9th, 2019

IBM Event Streams is IBM’s Kafka offering. Naturally it comes with it’s own UI and CLI tools, but one of the great things about Apache Kafka is that it’s not just a single thing from a single company – rather it is an active and diverse ecosystem, which means you’ve got a variety of tools to choose from.

I thought I’d try a couple of open source CLI tools, and share how to connect them and what they can do.

First up, kafkacat.

Read the rest of this entry »

Using nginx to enable custom domains in Auth0

December 2nd, 2018

In this post, I’m sharing the nginx config needed to set up a reverse proxy for enabling custom domains in Auth0.

I’ll start by explaining why I needed this, but if you don’t care about that, you can just skip straight to the code.

I’ve mentioned before that I use Auth0 for authentication and authorization in Machine Learning for Kids. (And I’ve mentioned before that the developer experience using Auth0 is generally fantastic).

But I’ve had one auth-related complaint from schools since launching the site. And it’s been increasing in the last couple of months:

Users who have their web browsers set to block third-party cookies weren’t able to log in.

This is because the cookies were coming from an auth0.com domain, and not machinelearningforkids.co.uk.

Read the rest of this entry »

How to create a Twitter API proxy using nginx in Cloud Foundry

November 18th, 2018

In this post, I’ll describe how to run nginx in Cloud Foundry to provide a Twitter API proxy that includes authentication and caching.

First, I want to talk a bit about why I wanted this, but if you don’t care about any of that, you can just skip to the code at the end of the post. 🙂

I’ve wanted for a while to enable projects in Machine Learning for Kids that use tweets. Using live tweets is a great way to make text analytics real for students, and a good example of how natural language processing is used in the real world.

The question was how to enable this from Scratch in a way that would be easy to use by schools.

The title of this post gives away the answer I ended up with, but I’ll describe why.

Read the rest of this entry »

Setting up Slack alerts to monitor IBM Event Streams

October 7th, 2018

IBM Event Streams brings Apache Kafka to IBM Cloud Private (together with a bunch of other useful stuff to make it easier to run and use Kafka).

Monitoring is an important part of running a Kafka cluster. There are a variety of metrics that are useful indicators of the health of the cluster and serve as warnings of potential future problems.

To that end, Event Streams collects metrics from all of the Kafka brokers and exports them to a Prometheus-based monitoring platform.

There are three ways to use this:

1) A selection of metrics can be viewed from a dashboard in the Event Streams admin UI.
eventstreams-monitoring-20181006-11
This is good for a quick way to get started.

2) Grafana is pre-configured and available out-of-the-box to create custom dashboards
eventstreams-monitoring-20181006-13
This will be useful for long-term projects, as Grafana lets you create dashboards showing the metrics that are most important for your unique needs. A sample dashboard is included to help get you started.

3) Alerts can be created, so that metrics that meet predefined criteria can be used to push notifications to a variety of tools, like Slack, PagerDuty, HipChat, OpsGenie, email, and many, many more.
eventstreams-monitoring-20181006-20
This is useful for being able to respond to changes in the metrics values when you’re not looking at the Monitor UI or Grafana dashboard.

For example, you might want a combination of alert approaches like:

  • metrics and/or metric values that might not be urgent but should get some attention result in an automated email being sent to a team email address
  • metrics and/or metric values that suggest a more severe issue could result in a Slack message to a team workspace
  • metrics and/or metric values that suggest an urgent critical issue could result in creating a PagerDuty ticket so that it gets immediate attention

This post is about this third use of monitoring and metrics: how you can configure alerts based on the metrics available from your Kafka brokers in IBM Event Streams.

Read the rest of this entry »