{"id":4201,"date":"2020-11-19T17:39:56","date_gmt":"2020-11-19T17:39:56","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4201"},"modified":"2020-11-19T17:46:24","modified_gmt":"2020-11-19T17:46:24","slug":"running-tensorflow-models-in-scratch","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4201","title":{"rendered":"Running TensorFlow models in Scratch"},"content":{"rendered":"<p><strong>I gave a short presentation today to explain how you can use TensorFlow machine learning models in the student block-based coding platform, Scratch.<\/strong><\/p>\n<p><strong>This post has the recording of my presentation, and I&#8217;ve put some notes (<em>all the stuff I meant to say but forgot!<\/em>) and screenshots below.<\/strong><\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/qHKwtefn21w\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><br \/>\n<small><a href=\"https:\/\/youtu.be\/qHKwtefn21w\">recording at https:\/\/youtu.be\/qHKwtefn21w<\/a><\/small><\/p>\n<p>I demonstrated three things:<\/p>\n<ol>\n<li>Using your own TensorFlow models in Scratch<\/li>\n<li>Using pretrained models in Scratch<\/li>\n<li>Creating TensorFlow models in <a href=\"https:\/\/teachablemachine.withgoogle.com\/\">Teachable Machine<\/a> and using them in Scratch<\/li>\n<\/ol>\n<p><!--more--><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-pretrained.png\"\/><\/p>\n<h3>If you know how to create your own models&#8230;<\/h3>\n<p>If you know some Python, you can build your own model. You can create simple tiny models. Or big complex models that take hours to train.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-tensorflowjs-converter.png\"\/><\/p>\n<p>Either way, once you have a model, you can use <a href=\"https:\/\/github.com\/tensorflow\/tfjs\/tree\/master\/tfjs-converter\">TensorFlow.js converter<\/a> to convert your model to a form that can be used in a web browser.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-imagenet-model.png\"\/><\/p>\n<p>Once converted, you need to host it somewhere online that Scratch will be able to access (<em>which means a server that won&#8217;t complain about CORS, if you know what that means<\/em>).<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-imagenet-open.png\"\/><\/p>\n<p>The <a href=\"https:\/\/machinelearningforkids.co.uk\/pretrained\">Machine Learning for Kids website<\/a> has a tool for launching Scratch with TensorFlow models &#8211; you just need to provide the URL for where your model is hosted.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-imagenet-testing.png\"\/><\/p>\n<p>I demonstrated it with a random imagenet model I found &#8211; using the machine learning model from Scratch to recognize the object in photos.<\/p>\n<p><em>It actually did better than I thought at the time. It classified a photo of a cat as a &#8220;tiger cat&#8221; which I thought was a mistake while I was demo&#8217;ing, but apparently that&#8217;s a way of describing some types of tabby cat.<\/em><\/p>\n<p>The point is, if you know how to create custom, bespoke machine learning models in TensorFlow, it&#8217;s simple to make them available for students to play with in Scratch.<\/p>\n<h3>If you can find pretrained models&#8230;<\/h3>\n<p>I didn&#8217;t actually train the imagenet model that I demonstrated myself. I got it from <a href=\"https:\/\/tfhub.dev\/s?deployment-format=tfjs&amp;module-type=image-classification\">TensorFlow Hub<\/a>.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-tfhub.png\"\/><\/p>\n<p>That was the second point in my talk &#8211; that if you don&#8217;t mind poking around sites like GitHub and TensorFlow Hub, then there are loads of interesting models around to try.<\/p>\n<p>Not all TF models will work with the Scratch plugin in Machine Learning for Kids. It&#8217;s very much an early, experimental feature. I&#8217;ve started with support for some types of image models, but I&#8217;m working on broadening it out to support more model types which should be available soon.<\/p>\n<h3>If you can make TensorFlow models in Teachable Machine&#8230;<\/h3>\n<p>This was the final demo, to make sure that everyone had at least something they could try.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-teachablemachine-training.png\"\/><\/p>\n<p>I used <a href=\"https:\/\/teachablemachine.withgoogle.com\/\">Teachable Machine<\/a> to train an image classifier to recognize a few Doctor Who minifigs I have on my desk.<\/p>\n<p>My plan was to train it to recognize all of them, but that took a lot longer than I expected so I ended up doing just a few to show the idea.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-teachablemachine-upload.png\"\/><\/p>\n<p>The nice thing about Teachable Machine is that not only will it export TensorFlow models for you, but that <a href=\"https:\/\/teachablemachine.withgoogle.com\/faq#Saving-&amp;-Exporting\">Google will host them<\/a> for you.<\/p>\n<p>This is something that <a href=\"https:\/\/github.com\/bonastreyair\/node-red-contrib-teachable-machine\">Node-RED has had support for<\/a> for ages, which is partly why I wanted to try doing something similar in Scratch (which is sort of like <a href=\"https:\/\/nodered.org\/\">Node-RED<\/a> but for kids!).<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-doctor-testing-1.png\"\/><\/p>\n<p>I did a quick bit of live-coding demo to show how you could use this to get a Scratch project to recognize the Doctor Who minifig visible in the webcam.<\/p>\n<p>Amazingly I managed to get through the whole demo without noticing that I&#8217;d left the webcam flipped, so my &#8220;clever&#8221; attempt at adding a relevant background was a little bit spoiled. Woops.<\/p>\n<p>I got a bit distracted playing with trying to get my webcam to focus on the tiny minifigs, so I probably spent longer on that demo than I needed to. Just getting a Scratch script to show what is in the webcam is fairly boring &#8211; as it&#8217;s just recreating what is already shown in the Teachable Machine web page.<\/p>\n<p><img decoding=\"async\" style=\"border: thin black solid; margin-top: 5px; margin-bottom: 5px;\" src=\"\/\/dalelane.co.uk\/blog\/post-images\/201119-tf-scratch\/201119-doctor-testing-2.png\"\/><\/p>\n<p>The more interesting opportunities are in making projects that respond in some way to what the machine learning model has recognized.<\/p>\n<p>For example, while I was playing with the demo, I thought of adding blocks that could play different sound effects for each Doctor (if I&#8217;d planned ahead and thought of that ahead of the demo, I could&#8217;ve got some sound files ready!), or adding a TARDIS sprite with costumes for each design of TARDIS, that changes to match the Doctor that is visible.<\/p>\n<p>(<em>I really like that idea, but I expect I&#8217;ll get myself into copyright trouble if I try and write a <a href=\"https:\/\/machinelearningforkids.co.uk\/worksheets\">worksheet<\/a> based on it&#8230; shame &#8211; I might still make the project anyway!<\/em>)<\/p>\n<blockquote class=\"twitter-tweet\">\n<p lang=\"en\" dir=\"ltr\">I gave a short presentation on using <a href=\"https:\/\/twitter.com\/hashtag\/TensorFlow?src=hash&amp;ref_src=twsrc%5Etfw\">#TensorFlow<\/a> models in <a href=\"https:\/\/twitter.com\/hashtag\/Scratch?src=hash&amp;ref_src=twsrc%5Etfw\">#Scratch<\/a>. The recording includes the live demo of how I made this Doctor Who themed project<a href=\"https:\/\/t.co\/VFxaubxcLH\">https:\/\/t.co\/VFxaubxcLH<\/a> <a href=\"https:\/\/t.co\/RnyqPzgtGL\">pic.twitter.com\/RnyqPzgtGL<\/a><\/p>\n<p>\u2014 Dale Lane (@dalelane) <a href=\"https:\/\/twitter.com\/dalelane\/status\/1329480949923524609?ref_src=twsrc%5Etfw\">November 19, 2020<\/a><\/p><\/blockquote>\n<p><script async=\"\" src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I gave a short presentation today to explain how you can use TensorFlow machine learning models in the student block-based coding platform, Scratch. This post has the recording of my presentation, and I&#8217;ve put some notes (all the stuff I meant to say but forgot!) and screenshots below. recording at https:\/\/youtu.be\/qHKwtefn21w I demonstrated three things: [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[580,587,536,600],"class_list":["post-4201","post","type-post","status-publish","format-standard","hentry","category-tech","tag-machine-learning","tag-mlforkids-tech","tag-scratch","tag-tensorflow"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4201","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4201"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4201\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4201"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4201"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4201"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}