{"id":3611,"date":"2018-06-03T01:25:34","date_gmt":"2018-06-03T01:25:34","guid":{"rendered":"http:\/\/dalelane.co.uk\/blog\/?p=3611"},"modified":"2019-04-11T14:44:43","modified_gmt":"2019-04-11T14:44:43","slug":"running-a-multi-region-cloud-foundry-application-in-ibm-cloud","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=3611","title":{"rendered":"Running a multi-region Cloud Foundry application in IBM Cloud"},"content":{"rendered":"

A few technical details on how I’m implementing global load balancing to improve the availability of Machine Learning for Kids<\/a>.<\/strong><\/p>\n

This wasn’t a great week for Machine Learning for Kids<\/a>. I think the site was unavailable for a couple of days in total this week, spread across a few outages – the worst one lasting over twelve hours<\/a>. I know I’ve lost some users as a result – a few teachers \/ coding group leaders did email me to say (not at all unreasonably) that they can’t use a tool that they can’t rely on. <\/p>\n

I wrote in my last post<\/a> that I would be making changes to prevent this sort of thing from happening again. Now that I’ve done it, I thought it\u2019d be good to share a few details on how I did it. <\/p>\n

(I should say that I owe a thanks to a ton of people at work who helped me with this. Times like this – when you need to do something new in a hurry – make me appreciate being able to get help from people across IBM in a hurry by asking a few questions in Slack!)<\/em> <\/p>\n

To start with, a recap of how the site was running before this weekend:<\/strong><\/p>\n

\"machinelearningforkids-before\"\/<\/a><\/p>\n

The site is implemented as a Node.js application<\/a>. It was deployed into Cloud Foundry running in the US South (Dallas) region of IBM Cloud<\/a>. <\/p>\n

By default, there were three instances of the app running.
\nThis improved reliability (if anything bad happened to cause one instance to crash, there would be two other instances available to keep handling requests without downtime).
\nIt improved performance (requests were round-robined across the three instances, so they worked together to handle all the requests the site got).
\nIt also made sure that I kept the implementation stateless, as I built everything assuming that consecutive requests might not go to the same instance. <\/p>\n

I used an Auto Scaling service<\/a> to add extra instances if things got busy. If the memory usage stayed above 80% for longer than a blip, it would start up an extra instance of the Node.js app to help share the load. And it was configured to keep adding instances if the site got even busier still. <\/p>\n

And when things quietened down, and the memory usage dropped and stayed below 30%, the service would kill the extra instances, scaling it down as low as three. <\/p>\n

That’s how I’d been running the site until now. It meant that the site handled spikes and busy periods (most recently Scratch Day<\/a>, when the app was being used by kids all around the world). <\/p>\n

But… it meant that went things went wrong in the Dallas region, my site dropped off the Internet. <\/p>\n

It was time for the site to grow up a little. <\/p>\n

Now the deployment looks like this:<\/strong><\/p>\n

\"machinelearningforkids-after\"\/<\/p>\n

It’s still a multi-instance Node.js application, deployed to Cloud Foundry in IBM Cloud. <\/p>\n

But now it’s deployed to two regions – the US South (Dallas) region and the EU GB (London) region. <\/p>\n

And I’ve got a load balancer in front, directing the requests. <\/p>\n

The US South instance is still going to be the primary instance and, if all is well, serve all of the API requests. When things get busy, there is still the Auto Scaling service to spin up additional instances of the Node.js app in the US South cloud. <\/p>\n

But when things go wrong, if the US South instances can’t be contacted, then the load balancer will start sending requests to the EU GB instance instead. That is only running a single instance normally, as when things are running smoothly, it\u2019ll be idle. But the Auto Scaling service is running there, too – so it can quickly increase the number of instances to match the needs of the current workload. <\/p>\n

That first layer, in front of the two IBM Cloud regions, is also providing a cache for all the static resources. This means the Node.js applications won’t have to do much work to serve the HTML, CSS, JavaScript and images that make up the training tool and Scratch. I’m not sure I really needed that, as the site generally runs with pretty low CPU and memory even when busy, but it was trivial to add, so I added it anyway.<\/em> <\/p>\n

That’s the big picture summary… so a few quick pointers for how I set it up. <\/p>\n

I’m using IBM’s Cloud Internet Services<\/a>. The first step was creating one of those services from the IBM Cloud Catalog<\/a>. That gave me a couple of addresses of nameservers I could use. <\/p>\n

I bought the machinelearningforkids.co.uk domain from eukhost<\/a>, so I needed to go to my admin page in eukhost and update the nameservers with those two new cloud.ibm.com addresses. <\/p>\n

<\/p>\n

I deployed the Node.js applications to the two regions – to the Dallas and London regions. <\/p>\n

I gave the US South deployment the following routes:
\nmachinelearningforkids.co.uk
\nwww.machinelearningforkids.co.uk
\nmachinelearningforkids-ussouth.mybluemix.net<\/code><\/p>\n

<\/p>\n

I gave the EU GB deployment the following routes:
\nmachinelearningforkids.co.uk
\nwww.machinelearningforkids.co.uk
\nmachinelearningforkids-eugb.eu-gb.mybluemix.net<\/code> <\/p>\n

<\/p>\n

Both deployments have the normal\/externally-visible site routes, and they both have a region-specific “internal” route. <\/p>\n

Note that this means I needed to add the custom domain and it’s SSL cert to both regions. <\/p>\n

<\/p>\n

I also added the certificate to the Cloud Internet Services instance, and set the TLS setting to \u201cEnd to end (flexible)”<\/p>\n

<\/p>\n

The setup process in Cloud Internet Services looks like this.<\/p>\n

I defined a “Health Check” – based on GET request to an API endpoint that doesn’t do anything other than check the application is responsive. <\/p>\n

<\/p>\n

I created a couple of “Origin Pools” – one for the US, and one for Europe. <\/p>\n

<\/p>\n

The US origin pool points at the region-specific route for the US application deployment:
\nmachinelearningforkids-ussouth.mybluemix.net<\/code><\/p>\n

<\/p>\n

The Europe origin pool points at the region-specific route for the UK application deployment:
\nmachinelearningforkids-eugb.eu-gb.mybluemix.net<\/code><\/p>\n

<\/p>\n

I defined the “Load Balancer”, pointing to both of these origin pools. The order is defined to give priority to the US origin pool.
\nAnd the hostname is set to “www<\/code>“. <\/p>\n

<\/p>\n

Finally, I created a single DNS record. <\/p>\n

A CNAME record with the name “@<\/code>” and the value “www.machinelearningforkids.co.uk<\/code>“. This points all records at the www<\/code> load balancer. <\/p>\n

<\/p>\n

As I said, I also enabled caching. It doesn’t really help with reliability, but was simple enough to turn on. <\/p>\n

I just added a couple of Page Rules, identifying URLs where resources are safe to cache. I\u2019ve long had all the static resources served at addresses starting https:\/\/machinelearningforkids.co.uk\/static\/*<\/code> and every build adds a unique build timestamp in the resource paths, so it’s safe to cache everything under here forever. <\/p>\n

<\/p>\n

That’s pretty much everything. <\/p>\n

To sum it all up:<\/p>\n

\"machinelearningforkids-after-notes\"\/<\/a><\/p>\n

(1)<\/strong>
\nThe user’s web browser goes to https:\/\/machinelearningforkids.co.uk<\/a>
\nThe nameservers for this address are now delegated to IBM CIS. <\/p>\n

(2)<\/strong>
\nAny requests for static resources (CSS, images, JavaScript, HTML components) are returned from the CIS cache directly
\nAPI requests need to go to a Node.js application. The DNS routing directs the request to the www<\/code> load balancer<\/p>\n

(3)<\/strong>
\nThe www<\/code> load balancer uses the health check I defined to poll the two origin pools regularly.
\nIf it knows that the US South deployment is healthy and responsive, it directs the request to machinelearningforkids-ussouth.mybluemix.net<\/code><\/p>\n

(4)<\/strong>
\nThe US South cloud is registered to handle requests to machinelearningforkids-ussouth.mybluemix.net<\/code> because that route is in the application’s deployment manifest.yml file<\/p>\n

(5)<\/strong>
\nCloud Foundry directs the request to one of the Node.js app instances<\/p>\n

(6)<\/strong>
\nThe Auto Scaling service monitors the memory usage of all of the Node.js app instances, adding or removing instances to keep the memory within thresholds that I\u2019ve set.<\/p>\n

This means the site is still going to be able to handle spikes and busy periods. And it should now remain accessible in the event of regional outages. <\/p>\n","protected":false},"excerpt":{"rendered":"

A few technical details on how I’m implementing global load balancing to improve the availability of Machine Learning for Kids. This wasn’t a great week for Machine Learning for Kids. I think the site was unavailable for a couple of days in total this week, spread across a few outages – the worst one lasting […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[587],"class_list":["post-3611","post","type-post","status-publish","format-standard","hentry","category-tech","tag-mlforkids-tech"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3611","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3611"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3611\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3611"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3611"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3611"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}