Pretrained models in Machine Learning for Kids

I’ve started adding pretrained machine learning models to Machine Learning for Kids. In this post, I wanted to describe what I’m doing.


Background

My basic approach with Machine Learning for Kids has been the same since I started it a few years ago.

Students:

  1. collect training examples
  2. train a model
  3. make something with that model

My development on the site over the last few years has largely been variations on that theme. Adding different ways of collecting training examples. Adding different types of machine learning model to train. Adding different ways to be able to make something with the models.

But the basic idea was still to let children learn about machine learning by creating their own model and making something with it.

That does have limitations though. There are many machine learning projects that would be fun to make, that aren’t realistic for a class of school children to train in a lesson – either because the labelling involved would be too complex, or the amount of training examples needed would be prohibitively time-consuming.

Pre-trained models

In the last few weeks, I’ve been adding a new section to Machine Learning for Kids to help to address that limitation: pretrained machine learning models.

I’ve started collecting some well-known machine learning models, integrating them into Scratch, and writing worksheets to help students get started with them.

Speech to text

This was the first one I added. I’ve added blocks that let you do speech-to-text in Scratch. You can record a few seconds of speech, and I’ve added a block that returns the text that was recognized.

I used the Web Speech API so this is all handled by the browser. The plus side of doing this meant it was trivial to implement. The down side is that the only browser that supports the Web Speech API is Google Chrome.

Face detection

This one was a little more involved. I’m using face-api.js which is a convenient API wrapper around TensorFlow.js models.

It has a few choices for models, but I’ve gone for the smallest SSD Mobilenet V1 face detector. Scratch is already a fairly heavyweight web app for the sorts of computers used in many schools, so keeping the stuff I add on top as small as possible is important.

The training dataset used for the model is WIDER FACE – a face detection benchmark dataset, made up of over 32,000 images labelled with the locations of over 390,000 faces.

My Scratch extension uses this model to detect a single face in what is currently visible on the Scratch stage, and adds a block that returns the x,y coordinates (translated into Scratch coordinates) for different face landmarks (eyes, nose, mouth).

I’ve intentionally not tied it to the webcam, as this way you can also detect faces in costumes, sprites, and on the background.

The video sensing extension built into Scratch adds a webcam feed to the stage, so if you want to do live face tracking on a video feed, I’ve made sure that my Scratch blocks work well with that, too.

Pose detection

This is my most recent addition. I’m using the posenet model from TensorFlow.js for this.

The training dataset for this is COCO (Common Objects in Context), and there is a good description of how it work on the TensorFlow blog.

My Scratch extension uses this model to detect a single person in what is currently visible on the Scratch stage, and adds a block that returns the x,y coordinates (translated into Scratch coordinates) for different body keypoints (wrists, elbows, shoulders, hips, knees, ankles).

As with the face detection extension, it’s not tied to the webcam, so although you can use it with a live webcam video feed, it works just as well with sprites and backgrounds.

Worksheets

I often find that features I add to the site aren’t heavily used until I write example worksheets to show what they can do. With that in mind, I’ve written a range of worksheets that demonstrate how to use these new pretrained models in Scratch projects.

Here are a few examples (with thanks to Faith for being my tolerant guinea pig as I’ve fleshed out these projects!).

Was this a good idea?

I agonised over doing this for ages. I still have some lingering doubts.

I’m not sure how much students will learn from using a pre-trained machine learning model as a black box without any insight into what it’s doing or how it was trained.

There are some ways I think teachers can get some educational benefit from them:

  • Encouraging students to read and think about how the models work
    Other than the speech-to-text model (because I don’t know how that works, as it’s baked into Google Chrome), I’m including links to places where these models are described. There is some opportunity for learning here, even if it’s more theory than hands-on.
  • Demonstrating the impact that the size of a training dataset has
    It’s useful to encourage students to compare what is possible with their own dataset of fewer-than-100 examples, with these models trained with 10,000s or 100,000s of training examples.
  • Demonstrating what machine learning can do
    It lets them see for themselves the range of what is possible today, without being restricted to what they can make themselves.
  • Illustrating how real-world machine learning projects work
    I think this is a really important point. Real-world and commercial ML projects will often use pre-trained models, because it is much quicker than creating a custom model and is still good enough for some tasks. Students can compare the experience of making a project where they train their own model with taking an off-the-shelf one, and it’s a useful insight into how ML projects are approached in industry.
  • Complementing a custom model with a pre-trained one
    In some of the worksheets, I’ve paired a custom model with a pre-trained one. For example, I’ve written a chatbot project worksheet where you train a custom text classifier to recognise the intent of a question, but use a pre-trained speech-to-text model to make a project that can answer spoken questions. I think this sort of blend could be the best of both worlds – some insight into training your own model, while getting the benefit of a powerful complex pre-trained model, all in one project.

But in the end, my rationale was mainly that they’re fun to use, and sometimes that is enough.

The projects where you train your own model will continue to be the primary focus of the site, but this just lets me supplement them with some fun projects that would be hard to make otherwise.

What next?

I’d like to add a few more models to this section of the site. If you’ve got any ideas of good ones to try, please let me know.

Tags:

This entry was posted on Monday, May 25th, 2020 at 3:03 pm and is filed under code. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

2 Responses to “Pretrained models in Machine Learning for Kids”

  1. Pedro says:
    Monday 17th August 2020 at 11:40 am

    Hi, the facefinder projects is not working for me, I see the sprites (eyes and nose) move when I move my face but they dont set on my own face, is like the face detection is not working, where can I find some help? thanks

  2. dale says:
    Thursday 20th August 2020 at 4:02 pm

    Hi Pedro. I replied to your forum post so I’m just waiting for you to provide an example I could use to investigate. I think that’s a better place than blog comments to track this.