{"id":4088,"date":"2020-08-20T19:34:39","date_gmt":"2020-08-20T19:34:39","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4088"},"modified":"2020-08-20T19:51:46","modified_gmt":"2020-08-20T19:51:46","slug":"supporting-ci-cd-with-kubernetes-operators","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4088","title":{"rendered":"Supporting CI\/CD with Kubernetes Operators"},"content":{"rendered":"<p><strong>Operators bring a lot of benefits as a way of managing complex software systems in a Kubernetes cluster. In this post, I want to illustrate one in particular: the way that custom resources (and declarative approaches to managing systems in general) enable easy integration with source control and a CI\/CD pipeline.<\/strong><\/p>\n<p><a target=\"_blank\" href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-cicd-overview.png\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-cicd-overview.png\" style=\"border: thin black solid;\"\/><\/a><\/p>\n<p>I&#8217;ll be using <a href=\"https:\/\/dalelane.co.uk\/blog\/?p=4066\">IBM Event Streams<\/a> as my example here, but the same principles will be true for many Kubernetes Operators, in particular, the open-source <a href=\"https:\/\/strimzi.io\/\">Strimzi Kafka Operator<\/a> that Event Streams is based on.<\/p>\n<p><!--more--><em>I&#8217;ve got a few screen recordings showing exactly how to set up a demo of this, which I&#8217;ve embedded at the start of each section. The screenshots underneath all link to a specific timestamp of the relevant video.<\/em><\/p>\n<h3>The developer example<\/h3>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/fEay4Qvj7RY\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY\" rel=\"noopener noreferrer\">video<\/a><\/em><\/small><\/p>\n<p>Imagine you&#8217;re a developer. You&#8217;re working on a Java application.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=24\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_00_24.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=24\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>The details of the app aren&#8217;t important for the purposes of this demo, but I forked and tweaked the awesome <a href=\"https:\/\/github.com\/ibm-messaging\/kafka-java-vertx-starter\" target=\"_blank\" rel=\"noopener noreferrer\">kafka-java-vertx-starter<\/a> app as a basis for showing what&#8217;s possible, as it&#8217;s a simple Kafka Java app with a nice UI.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=14\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_00_14.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=14\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>There are two instances of Event Streams here &#8211; two Kafka clusters.<\/p>\n<ul>\n<li>A <strong>development<\/strong> instance &#8211; that you&#8217;re free to play with, make changes to and test things on<\/li>\n<li>A <strong>production<\/strong> instance &#8211; that is locked down, with all changes made in a controlled automated way<\/li>\n<\/ul>\n<p>Now <strong>you need a new Kafka topic<\/strong> for your application.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=42\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_00_42.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=42\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>You can use the Event Streams user interface for the development instance for this &#8211; giving you a friendly guided wizard to help you work out the right config for your topic.<\/p>\n<p>You can test your app against it, and tweak the config if you need to.<\/p>\n<p>Once you&#8217;re happy with it, it&#8217;s time to commit this.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=60\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_01_00.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=60\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Everything in Event Streams is represented under the covers as custom resources, which means you can get the custom resource for your new topic from the development instance.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=65\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_01_05.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=65\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Everything you chose in the UI &#8211; all the final options and config you settled on through interactively clicking through the UI wizard &#8211; is captured here by the Event Streams Operator.<\/p>\n<p>Kubernetes can give this to you in YAML or JSON form, but let&#8217;s go with YAML for now.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=93\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_01_33.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=93\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>You&#8217;ll do a tiny bit of editing: stripping it down to the spec by removing the status. You can make the spec more generic and flexible by replacing the Event Streams instance name and deployment namespace with environment variables.<\/p>\n<p>What you&#8217;re left with is a human-readable plain text file that describes the new topic you need for your application. This can live alongside the rest of the source code for your application, letting you manage them together.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=118\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_01_58.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=118\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>You can put this in a pull-request so your colleagues can review that the spec looks reasonable for promoting to the production cluster instance.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=138\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_02_19.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=138\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>(You could put the topic spec in the same pull-request as whatever code changes you made to your application that needed the new topic &#8211; so the reviewer can see why you need the new topic in the context of your application, but let&#8217;s just keep the pull-request simple with the single file for now).<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=150\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_02_30.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=150\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Once your pull-request is reviewed and approved, you can merge your branch into master.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=178\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_02_58.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=178\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>At that point, a Travis job will be started &#8211; taking your new custom resource spec requesting a new Kafka topic, and automatically applying it to the production Event Streams cluster instance.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=178\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_02_50.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=178\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>This means the Event Streams Operator managing the production cluster will automatically create your new topic, matching the attributes and config that you had working in the development environment.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/fEay4Qvj7RY?start=207\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY\" rel=\"noopener noreferrer\">video<\/a><\/em><\/small><\/p>\n<p>To help make this clear, let&#8217;s do that again. But this time, <strong>you need a new set of Kafka credentials<\/strong> for your application.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=220\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_03_40.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=220\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>As before, you do this using your development cluster, using the wizard in the Event Streams UI to interactively define credentials with the correct permissions for your application.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=254\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_04_14.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=254\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>The Event Streams Operator will capture all of this in the custom resource representing the credentials, so once you&#8217;re happy that everything is working, you can retrieve the spec from Kubernetes.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=268\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_04_28.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=268\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Again, you need to do a tiny bit of editing &#8211; removing the dynamic status sections to get just the specification, and replacing the cluster and namespace names with environment variables to make it easier to apply to other clusters.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=309\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_05_09.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=309\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>As before, you&#8217;re left with a human-readable plain text file that describes what your application needs. This sort of YAML specification might be a bit tricky for a novice user to write from scratch but the admin UI did that for you as a result of clicking through the wizard.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=340\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_05_40.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=340\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Putting the resulting specification in a pull-request makes it easy for a reviewer to read and follow what is being asked for.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=366\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_06_06.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=366\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Once this pull-request is approved and merged, Travis runs again, applying the latest change to the production cluster, so the new approved set of credentials are automatically provisioned on the production Event Streams cluster by the Event Streams Operator, with permissions matching what was verified in the development environment.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=405\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-developer-00_06_45.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/fEay4Qvj7RY?t=405\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<h3>The admin\/ops example<\/h3>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/jWQL0QlKoHY\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY\" rel=\"noopener noreferrer\">video<\/a><\/em><\/small><\/p>\n<p>This sort of workflow isn&#8217;t just useful for developers. Cluster administrators can use the same approach for managing the deployment of the whole Kafka cluster.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=17\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_00_17.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=17\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>As with everything else in Event Streams, the specification for the Kafka cluster is defined in a Kubernetes custom resource, that you can retrieve in JSON or YAML form.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=51\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_00_51.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=51\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>For this example, let&#8217;s say you want to make a couple of changes. <strong>You want to scale up your Kafka cluster<\/strong> from three brokers to four, so you update the replicas attribute.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=61\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_01_01.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=61\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>And <strong>you want to lock down the broker to prevent topics from being automatically created<\/strong>, so you add that to the Kafka cluster config.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=80\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_01_20.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=80\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>You start by applying this to the development cluster, to make sure that you&#8217;re happy with the change.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=152\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_02_32.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=152\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p><em>This takes a little while, so if you&#8217;re following the video, you might want to fast-forward through this bit! <\/em><\/p>\n<p><em>It&#8217;s just watching-and-waiting &#8211; there isn&#8217;t anything for the administrator to do as the Event Streams Operator takes care of orchestrating the careful roll of the cluster to apply the changes you&#8217;ve requested, without any disruption or downtime.<\/em><\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=268\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_04_28.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=268\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Just like the Kafka topic and client credentials, once you&#8217;re happy you can put the YAML specification for the cluster into a GitHub pull-request for review and approval.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=285\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_04_45.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=285\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>As you can see from this, this means you&#8217;re not restricted to creating new things in these changes.<\/p>\n<p>You can use pull-requests to modify the spec for existing things &#8211; changing the config for a topic, modifying the permissions for client credentials, and so on.<\/p>\n<p>Every time you want to make a change, that&#8217;s another edit to a YAML file in another pull request. And GitHub will keep a full history of all of these changes.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=326\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-admin-00_05_26.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/jWQL0QlKoHY?t=326\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Once the pull-request is merged, Travis will apply the updated spec to the production cluster, and the Event Streams Operator will start rolling that change out to the production instance &#8211; updating the Kafka cluster config and scaling up the cluster with an additional broker.<\/p>\n<h3>Where all of this is happening<\/h3>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/cD5EE3V87p8\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/cD5EE3V87p8\" rel=\"noopener noreferrer\">video<\/a><\/em><\/small><\/p>\n<p>As you can see, it&#8217;s easy to retrieve the specifications for all aspects of your Kafka cluster from Event Streams. And because they&#8217;re just plain text, they&#8217;re easy to store and track in source control systems like GitHub.<\/p>\n<p>The only bit I haven&#8217;t shown is how these specifications are being magically being applied to the production cluster.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/cD5EE3V87p8?t=6\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-travis-00_00_06.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/cD5EE3V87p8?t=6\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>As I mentioned, I used Travis for this, but there are loads of ways to trigger jobs from GitHub so this by no means the only way I could&#8217;ve done it.<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-travis-settings.png\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-travis-settings.png\" style=\"border: thin black solid;\"\/><\/a><\/p>\n<p>I needed to give Travis the details for my production Event Streams instance, so I did this using a few environment variables:<\/p>\n<ul>\n<li><code>KUBECONFIG<\/code> &#8211; with the credentials and address of my production Kubernetes cluster<\/li>\n<li><code>TARGET_CLUSTER_NAME<\/code> &#8211; with the name of my Event Streams instance<\/li>\n<li><code>TARGET_NAMESPACE<\/code> &#8211; with the name of the Kubernetes namespace where it is running<\/li>\n<\/ul>\n<p>To get a convenient value for <code>KUBECONFIG<\/code>, I collapsed all of this down into a single simple string by base64-encoding the contents of my kubectl config.<\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">kubectl config view | base64<\/pre>\n<p><a target=\"_blank\" href=\"https:\/\/youtu.be\/cD5EE3V87p8?t=29\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-travis-00_00_29.png\" style=\"border: thin black solid;\"\/><\/a><br \/>\n<small><em><a target=\"_blank\" href=\"https:\/\/youtu.be\/cD5EE3V87p8?t=29\" rel=\"noopener noreferrer\">jump to screen-recording<\/a><\/em><\/small><\/p>\n<p>Here is the Travis job I&#8217;m using. I&#8217;ve added comments to each section so that you can follow what it&#8217;s doing:<\/p>\n<ul>\n<li>download kubectl<\/li>\n<li>configure it with my <code>KUBECONFIG<\/code><\/li>\n<li>edit the YAML specs to substitute in the target Event Streams instance details<\/li>\n<li>apply the YAML specs to the production cluster<\/li>\n<\/ul>\n<p>Putting all of the Kubernetes specs in the same folder means the Travis job doesn&#8217;t need updating every time a new spec is added.<\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em; overflow-x: scroll; font-size: 1.1em;\"><font color=\"#006600\"># I just need curl to be able to download kubectl<\/font>\n<font color=\"#660000\">language<\/font>: <font color=\"#0000FF\">minimal<\/font>\n\n<font color=\"#006600\"># I want to run this job when pull-requests are<\/font>\n<font color=\"#006600\">#  merged into the master branch<\/font>\n<font color=\"#660000\">branches<\/font>:\n  <font color=\"#660000\">only<\/font>:\n  - <font color=\"#0000FF\">master<\/font>\n\n<font color=\"#006600\"># The dependency for this travis job is kubectl<\/font>\n<font color=\"#006600\">#  I copied these script steps for installing it from<\/font>\n<font color=\"#006600\">#  https:\/\/kubernetes.io\/docs\/tasks\/tools\/install-kubectl\/<\/font>\n<font color=\"#660000\">install<\/font>:\n  - <font color=\"#0000FF\">curl -LO \"https:\/\/storage.googleapis.com\/kubernetes-release\/release\/$(curl -s https:\/\/storage.googleapis.com\/kubernetes-release\/release\/stable.txt)\/bin\/linux\/amd64\/kubectl\"<\/font>\n  - <font color=\"#0000FF\">chmod +x .\/kubectl<\/font>\n  - <font color=\"#0000FF\">sudo mv .\/kubectl \/usr\/local\/bin\/kubectl<\/font>\n\n<font color=\"#660000\">before_script:<\/font>\n  <font color=\"#006600\"># The credentials for my Kubernetes cluster is managed<\/font>\n  <font color=\"#006600\">#  by Travis in a secure environment variable<\/font>\n  <font color=\"#006600\"># I need to copy that into a file kubectl can use<\/font>\n  - <font color=\"#0000FF\">echo \"$KUBECONFIG\" | base64 --decode &gt; kubeconfig<\/font>\n  <font color=\"#006600\"># The target cluster is identified by Travis using<\/font>\n  <font color=\"#006600\">#  environment variables, so I need to modify the<\/font>\n  <font color=\"#006600\">#  specification files to use them<\/font>\n  - <font color=\"#0000FF\">sed -i \"s\/TARGET_CLUSTER_NAME\/$TARGET_CLUSTER_NAME\/g\" kafka-resources\/*<\/font>\n  - <font color=\"#0000FF\">sed -i \"s\/TARGET_NAMESPACE\/$TARGET_NAMESPACE\/g\"       kafka-resources\/*<\/font>\n\n<font color=\"#006600\"># Time to apply the latest Kubernetes resources.<\/font>\n<font color=\"#006600\"># Not all commits to the master branch will have edited<\/font>\n<font color=\"#006600\">#  these resources, but Kubernetes is smart enough to<\/font>\n<font color=\"#006600\">#  ignore requests that don't change anything.<\/font>\n<font color=\"#660000\">script<\/font>:\n  - <font color=\"#0000FF\">kubectl --kubeconfig=kubeconfig apply -f kafka-resources\/<\/font>\n\n<\/pre>\n<h3>What have we done?<\/h3>\n<p>This has all been a long-winded way to demonstrate this:<\/p>\n<p><a target=\"_blank\" href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-cicd-overview.png\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/200820-operator-cicd\/200820-cicd-overview.png\" style=\"border: thin black solid;\"\/><\/a><\/p>\n<p>Developers get a guided experience in an admin UI, helping them to interactively define what they need in a development cluster.<\/p>\n<p>Once they&#8217;re happy with it, they retrieve the resulting specifications and put them in GitHub for review.<\/p>\n<p>Once the changes are approved, Travis applies them to the production cluster.<\/p>\n<h3>Why is this useful?<\/h3>\n<p>This has many benefits, but this post is already longer than I planned, so I won&#8217;t go into them here. If you do a web search for &#8220;configuration as code&#8221; you can find a variety of compelling arguments for why this approach can be great. You get the workflow benefits of an automated rollout of tested and reviewed changes to your production system, a version history for all config changes you make (captured together with and in the context of the related application logic changes), and much more.<\/p>\n<h3>What else could I have done?<\/h3>\n<p>Loads!<\/p>\n<p>For example, I could&#8217;ve used another GitHub branch to <strong>support a staging\/test Event Streams instance in between development and production<\/strong>. For pull-requests into a <code>develop<\/code> branch, Travis could apply those changes to a test Event Streams instance. And then only apply the specs to the production Event Streams instance when the changes are promoted to the <code>master<\/code> branch.<\/p>\n<p>The Travis job I used will create new things and modify existing things, but I could have <strong>extended the Travis job to support deleting things<\/strong> as well:<\/p>\n<ul>\n<li>Adding a folder called <code>deleted-kafka-resources<\/code> to the application repo<\/li>\n<li>Adding <code>kubectl delete -f deleted-kafka-resources\/<\/code> to my Travis script<\/li>\n<\/ul>\n<p>This would allow developers to delete things in a similar way.<\/p>\n<p>For example, a pull-request that moves the topic.yaml file from the kafka-resources folder to the deleted-kafka-resources folder would automatically delete the Kafka topic on the production Event Streams instance once it&#8217;s merged.<\/p>\n<p>I&#8217;ve described testing as a manual step against the development or test Event Streams instance, but <strong>automated testing of configuration changes could be brought into Travis<\/strong>, too.<\/p>\n<p>If you have integration tests for your Kafka application, then these could be run in a Travis job for your pull-request:<\/p>\n<ul>\n<li>Applying the updated custom-resource specifications to a test Event Streams cluster instance<\/li>\n<li>Running the integration tests against that modified Event Streams instance<\/li>\n<\/ul>\n<p>The results from these tests could be made an automated check for the pull-request, giving the reviewer an extra bit of confirmation that the proposed configuration changes have been verified.<\/p>\n<p>And lots more.<\/p>\n<p>As I said at the start, a lot of what I&#8217;ve covered here will be true for many Operators, and you could do a very similar demo to what I&#8217;ve done here using the open source <a href=\"https:\/\/strimzi.io\/\">Strimzi Kafka Operator<\/a> that Event Streams is based on.<\/p>\n<p>The important point is that the declarative approach to managing complex systems that is enabled by Kubernetes Operators makes it easy to integrate with a broader CI\/CD strategy.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Operators bring a lot of benefits as a way of managing complex software systems in a Kubernetes cluster. In this post, I want to illustrate one in particular: the way that custom resources (and declarative approaches to managing systems in general) enable easy integration with source control and a CI\/CD pipeline. I&#8217;ll be using IBM [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":4111,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,582,583,584,599],"class_list":["post-4088","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-apachekafka","tag-eventstreams","tag-ibmeventstreams","tag-kafka","tag-kubernetes"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4088","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4088"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4088\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/4111"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4088"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4088"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4088"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}