{"id":4639,"date":"2022-07-19T17:16:54","date_gmt":"2022-07-19T17:16:54","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4639"},"modified":"2022-07-23T01:52:26","modified_gmt":"2022-07-23T01:52:26","slug":"how-to-scale-ibm-mq-clusters-and-client-applications-in-openshift","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4639","title":{"rendered":"How to scale IBM MQ clusters and client applications in OpenShift"},"content":{"rendered":"<h2 id=\"summary_overview\">Overview<\/h2>\n<p>You&#8217;re running a cluster of IBM MQ queue managers in Red Hat OpenShift, together with a large number of client applications putting and getting messages to them. This workload will vary over time, so you need flexibility in how you scale all of this.<\/p>\n<p>This tutorial will show how you can easily scale the number of instances of your client applications up and down, without having to reconfigure their connection details and without needing to manually distribute or load balance them.<\/p>\n<p>And it will show how to quickly and easily grow the queue manager cluster &#8211; adding a new queue manager to the cluster without complex, new, custom configuration.<\/p>\n<h2 id=\"summary_background\">Background<\/h2>\n<p>The IBM MQ feature demonstrated in this tutorial is <strong>Uniform Clusters<\/strong>. Dave Ware has a great <a href=\"https:\/\/youtu.be\/LWELgaEDGs0\">introduction and demo of Uniform Clusters<\/a>, so if you&#8217;re looking for background about how the feature works, I&#8217;d highly recommend it.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/LWELgaEDGs0\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p>This tutorial is heavily inspired by that demo (<em>thanks, Dave!<\/em>), but my focus here is mainly on how to apply the techniques that Dave showed in OpenShift.<\/p>\n<p><!--more--><\/p>\n<div class=\"mqunicontents\">\n<ul>\n<li><a href=\"#demo_prereqs\">Pre-requisites<\/a><\/li>\n<li><a href=\"#demo_mqsetup\">Creating an IBM MQ Uniform Cluster in OpenShift<\/a><\/li>\n<li>Load balancing client applications\n<ul>\n<li><a href=\"#demo_jmssetup\">Building JMS client applications to run in OpenShift<\/a><\/li>\n<li><a href=\"#demo_config_apps\">Configuring JMS client applications in OpenShift<\/a><\/li>\n<li><a href=\"#demo_scaleup_apps\">Load balancing when client applications are scaled up<\/a><\/li>\n<li><a href=\"#demo_scaledown_apps\">Load balancing when client applications are scaled down<\/a><\/li>\n<li><a href=\"#demo_scaletoolow\">Load balancing if there aren&#8217;t enough client applications<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"#demo_scale_qmgrs\">Scaling the queue manager cluster<\/a><\/li>\n<\/ul>\n<\/div>\n<h1 id=\"demo\">Demo<\/h1>\n<p><!--\n    DEMO VIDEO timings\n\n    01-setup.sh\n    00:00-00:24 snip 00:24-00:37\n\n    02-build-apps.sh\n    00:39-00:59\n\n    cd demo-scripts\n    start-side-bar\n    01:00-01:10\n\n    deploy-test-apps-start-with-qm1.sh\n    01:11-01:58\n    21\/0\/0 at 01:32\n    7\/7\/7 at 01:52\n\n    delete-test-apps.sh\n    02:00-02:18\n\n    deploy-test-apps-randomly-distributed.sh\n    02:19-03:07\n\n    delete-test-apps.sh\n    03:08-03:27\n\n    deploy-test-apps-randomly-distributed.sh\n    03:28-04:03\n    6\/4\/11 at 03:44\n    7\/7\/7 at 04:01\n\n    delete-test-apps.sh\n    04:04-04:17\n\n    deploy-test-apps-start-with-qm1.sh\n    04:19-04:52\n    21\/0\/0 at 04:32\n    7\/7\/7 at 04:46\n\n    scale-up-getter.sh\n    04:52-05:37\n    16\/7\/7 at 05:04\n    10\/10\/10 at 05:32\n\n    scale-one-per-qmgr.sh\n    05:38-06:10\n    2\/0\/1 at 05:57\n    1\/1\/1 at 06:02\n\n    scale-one-only.sh\n    06:13-06:38\n    1\/0\/0 at 06:22\n\n    scale-up-getter.sh\n    06:39-07:29\n    10\/10\/10 at 07:22\n\n    03-setup-additional-qmgr.sh\n    07:32-09:49\n    7\/8\/8\/7 at 09:39\n\n    04-delete-everything.sh\n    09:50-10:10\n--><\/p>\n<h2 class=\"mqunisection\" id=\"demo_prereqs\">Pre-requisites<\/h2>\n<p>The best way to understand how this all works is to set it up for yourself. If you want to create a demo, you can simply follow the instructions below.<\/p>\n<p>You will need:<\/p>\n<ul>\n<li>a Red Hat OpenShift cluster<\/li>\n<li>on your local machine:\n<ul>\n<li>the <a href=\"https:\/\/docs.openshift.com\/container-platform\/4.8\/cli_reference\/openshift_cli\/getting-started-cli.html\">OpenShift <code class=\"mqunicode\">oc<\/code> CLI<\/a>, logged into your cluster<\/li>\n<li><a href=\"https:\/\/formulae.brew.sh\/formula\/gnu-sed\"><code class=\"mqunicode\">gsed<\/code><\/a><\/li>\n<li><code class=\"mqunicode\">docker<\/code> CLI<\/li>\n<li><code class=\"mqunicode\">Java<\/code> for compiling and running the test applications<\/li>\n<li><code class=\"mqunicode\">mvn<\/code> for building the test applications<\/li>\n<li>a copy of the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\">mq-uniformclusters-openshift-tutorial<\/a> repository from Github<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>But if you would rather just see it in action, I&#8217;ll also include a description of how the setup works and screen-recording videos of the demo running.<\/p>\n<h2 class=\"mqunisection\" id=\"demo_mqsetup\">Creating an IBM MQ Uniform Cluster in OpenShift<\/h2>\n<pre class=\"mqunistep\">.\/01-setup.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc\">You can see this between 00:00 and 00:37 in the demo video<\/a><\/em><\/small><\/p>\n<p>The <code class=\"mqunicode\">01-setup.sh<\/code> demo script creates a queue manager uniform cluster with three queue managers. It looks like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/00-three-qmgrs.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p>There are a few elements to this.<\/p>\n<p>A <code class=\"mqunicode\">ConfigMap<\/code> with a <code class=\"mqunicode\">config.ini<\/code> is used by <strong>all three queue managers<\/strong>. This identifies which of the three queue managers are going to maintain the full repository of information about the cluster. In <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-resources\/configmap-uniformcluster.yaml\">this demo config<\/a>, QM1 and QM2 will be the full repositories.<\/p>\n<pre class=\"mqunilisting\">apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: mq-uniform-cluster-ini-cm\ndata:\n  config.ini: |-\n    AutoCluster:\n      Repository2Conname=uniform-cluster-qm1-ibm-mq.uniform-cluster.svc(1414)\n      Repository2Name=QM1\n      Repository1Conname=uniform-cluster-qm2-ibm-mq.uniform-cluster.svc(1414)\n      Repository1Name=QM2\n      ClusterName=DEMOCLUSTER\n      Type=Uniform<\/pre>\n<p>Another <code class=\"mqunicode\">ConfigMap<\/code> holds the MQSC commands that <strong>all three queue managers<\/strong> use to define the channel they will need to be members of the cluster. In <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-resources\/configmap-common.yaml\">this demo config<\/a>, that MQSC file is called <code class=\"mqunicode\">common_config.mqsc<\/code>.<\/p>\n<pre class=\"mqunilisting\">apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: mq-uniform-cluster-mqsc-cm\ndata:\n  common_config.mqsc: |-\n    define channel('+AUTOCL+_+QMNAME+') chltype(clusrcvr) trptype(tcp) conname(+CONNAME+) cluster('+AUTOCL+') replace<\/pre>\n<p>Each queue manager then has it&#8217;s own <code class=\"mqunicode\">ConfigMap<\/code> with an additional MQSC file defining the addresses for the channels it will use to join the cluster. For example, <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-resources\/configmap-qm3.yaml\">the config for QM3 in this demo<\/a> defines the cluster sender channels to the two full repositories, and the cluster receiver that QM3 uses to receive connections from other cluster members.<\/p>\n<pre class=\"mqunilisting\">apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: mq-uniform-cluster-qm3-mqsc-cm\ndata:\n  qm3-config.mqsc: |-\n    alter chl(DEMOCLUSTER_QM1) chltype(CLUSSDR) conname('uniform-cluster-qm1-ibm-mq.uniform-cluster.svc(1414)')\n    alter chl(DEMOCLUSTER_QM2) chltype(CLUSSDR) conname('uniform-cluster-qm2-ibm-mq.uniform-cluster.svc(1414)')\n    alter chl(DEMOCLUSTER_QM3) chltype(CLUSRCVR) conname('uniform-cluster-qm3-ibm-mq.uniform-cluster.svc(1414)')<\/pre>\n<p>The <code class=\"mqunicode\">QueueManager<\/code> specifications for each of the queue managers just need to point at these ConfigMaps. For example, the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-resources\/qmgr-qm3.yaml\">config for QM3 in this demo<\/a> starts with this:<\/p>\n<pre class=\"mqunilisting\">apiVersion: mq.ibm.com\/v1beta1\nkind: QueueManager\nmetadata:\n  name: uniform-cluster-qm3\nspec:\n  queueManager:\n    name: QM3\n    ini:\n      - configMap:\n          items:\n          - config.ini\n          name: mq-uniform-cluster-ini-cm\n    mqsc:\n      - configMap:\n          items:\n          - common_config.mqsc\n          name: mq-uniform-cluster-mqsc-cm\n      - configMap:\n          name: mq-uniform-cluster-qm3-mqsc-cm\n          items:\n            - qm3-config.mqsc<\/pre>\n<p>Creating all of this results in a uniform cluster running in OpenShift with three queue managers, as shown in the diagram above.<\/p>\n<h2 class=\"mqunisection\">Load balancing client applications<\/h2>\n<h2 id=\"demo_jmssetup\">Building JMS client apps to run in OpenShift<\/h2>\n<pre class=\"mqunistep\">.\/02-build-apps.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=62\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=39\">You can see this between 00:39 and 00:59 in the demo video<\/a><\/em><\/small><\/p>\n<p>A <code class=\"mqunicode\">Deployment<\/code> is created that runs a simple JMS application that <strong>puts<\/strong> messages to the <code class=\"mqunicode\">APPQ1<\/code> queue in the uniform cluster.<\/p>\n<p>A second <code class=\"mqunicode\">Deployment<\/code> is created to run a JMS application that <strong>gets<\/strong> messages from the <code class=\"mqunicode\">APPQ1<\/code> queue in the uniform cluster.<\/p>\n<p>To start with, the config in this demo runs twenty-one instances of the getter application.<\/p>\n<pre class=\"mqunilisting\">apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: test-app-getter\n  labels:\n    app: jms-getter\nspec:\n  replicas: 21\n  selector:\n    matchLabels:\n      app: jms-getter\n  template:\n    metadata:\n      labels:\n        app: jms-getter\n    spec:\n      volumes:\n        - name: ccdtfile\n          configMap:\n            name: ccdt\n      containers:\n          - volumeMounts:\n              - name: ccdtfile\n                mountPath: \/opt\/app\/config<\/pre>\n<p>This will create twenty-one Kubernetes pods, each one running a separate instance of the JMS application. All twenty-one instances of the application have a unique client id, however they set their App Name to the same value, so that IBM MQ recognises that they are all instances of the same application. You can see this in the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-jms-ccdt\/src\/main\/java\/com\/ibm\/clientengineering\/mq\/samples\/Getter.java\">Java source for the JMS application<\/a>.<\/p>\n<pre class=\"mqunilisting\">cf.setAppName(\"test-getter\");<\/pre>\n<p>All twenty-one instances of the applications also share a common set of connection information to IBM MQ, which is defined in a <code class=\"mqunicode\">ConfigMap<\/code>.<\/p>\n<p>In <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/test-app-resources\/ccdt-start-with-qm1-first.yaml\">this demo config<\/a>, the configmap is called &#8220;ccdt&#8221;, and contains connection information for:<\/p>\n<ul>\n<li>each queue manager as a member of the cluster (address of the queue manager, with a <code class=\"mqunicode\">queueManager<\/code> name of <code class=\"mqunicode\">DEMOCLUSTER<\/code>)<\/li>\n<li>each queue manager itself (address of the queue manager with it&#8217;s own <code class=\"mqunicode\">queueManager<\/code> name)<\/li>\n<\/ul>\n<pre class=\"mqunilisting\">kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: ccdt\ndata:\n  ibm-mq-ccdt.json: |-\n    {\n        \"channel\": [\n          {\n            \"name\": \"DEF.SVRCONN\",\n            \"clientConnection\": {\n              \"connection\": [\n                {\n                  \"host\": \"uniform-cluster-qm1-ibm-mq.uniform-cluster\",\n                  \"port\": 1414\n                }\n              ],\n              \"queueManager\": \"DEMOCLUSTER\"\n            },\n            \"transmissionSecurity\": {\n              \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n            },\n            \"type\": \"clientConnection\"\n          },\n          {\n            \"name\": \"DEF.SVRCONN\",\n            \"clientConnection\": {\n              \"connection\": [\n                {\n                  \"host\": \"uniform-cluster-qm2-ibm-mq.uniform-cluster\",\n                  \"port\": 1414\n                }\n              ],\n              \"queueManager\": \"DEMOCLUSTER\"\n            },\n            \"transmissionSecurity\": {\n              \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n            },\n            \"type\": \"clientConnection\"\n          },\n          {\n            \"name\": \"DEF.SVRCONN\",\n            \"clientConnection\": {\n              \"connection\": [\n                {\n                  \"host\": \"uniform-cluster-qm3-ibm-mq.uniform-cluster\",\n                  \"port\": 1414\n                }\n              ],\n              \"queueManager\": \"DEMOCLUSTER\"\n            },\n            \"transmissionSecurity\": {\n              \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n            },\n            \"type\": \"clientConnection\"\n          },\n          {\n            \"name\": \"DEF.SVRCONN\",\n            \"clientConnection\": {\n              \"connection\": [\n                {\n                  \"host\": \"uniform-cluster-qm1-ibm-mq.uniform-cluster\",\n                  \"port\": 1414\n                }\n              ],\n              \"queueManager\": \"QM1\"\n            },\n            \"transmissionSecurity\": {\n              \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n            },\n            \"type\": \"clientConnection\"\n          },\n          {\n            \"name\": \"DEF.SVRCONN\",\n            \"clientConnection\": {\n              \"connection\": [\n                {\n                  \"host\": \"uniform-cluster-qm2-ibm-mq.uniform-cluster\",\n                  \"port\": 1414\n                }\n              ],\n              \"queueManager\": \"QM2\"\n            },\n            \"transmissionSecurity\": {\n              \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n            },\n            \"type\": \"clientConnection\"\n          },\n          {\n            \"name\": \"DEF.SVRCONN\",\n            \"clientConnection\": {\n              \"connection\": [\n                {\n                  \"host\": \"uniform-cluster-qm3-ibm-mq.uniform-cluster\",\n                  \"port\": 1414\n                }\n              ],\n              \"queueManager\": \"QM3\"\n            },\n            \"transmissionSecurity\": {\n              \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n            },\n            \"type\": \"clientConnection\"\n          }\n        ]\n      }<\/pre>\n<p>The volume mount of the ccdt ConfigMap in the <code class=\"mqunicode\">Deployment<\/code> means that the CCDT JSON is available in the Kubernetes pod as a file. This means it is easy for the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-jms-ccdt\/src\/main\/java\/com\/ibm\/clientengineering\/mq\/samples\/Getter.java\">JMS application to get the MQ connection config<\/a> from the ConfigMap.<\/p>\n<pre class=\"mqunilisting\">public static final String CCDT_LOCATION = \"\/opt\/app\/ibm-mq-ccdt.json\";\n\nFile ccdtfile = new File(Config.CCDT_LOCATION);\nMQConnectionFactory cf = new MQConnectionFactory();\ncf.setCCDTURL(ccdtfile.toURI().toURL());<\/pre>\n<h2 id=\"demo_config_apps\">Configuring JMS client apps in OpenShift<\/h2>\n<pre class=\"mqunistep\">.\/demo-scripts\/deploy-test-apps-start-with-qm1.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=71\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=71\">You can see this between 01:11 and 01:58 in the demo video<\/a><\/em><\/small><\/p>\n<p>The <code class=\"mqunicode\">deploy-test-apps-start-with-qm1.sh<\/code> script shows how you can choose to configure client apps to initially connect to the first queue manager in <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/test-app-resources\/ccdt-start-with-qm1-first.yaml\">the CCDT list<\/a>.<\/p>\n<p>If you choose to do this, as you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=92\">the demo video<\/a> (at approx 1min 32secs) all twenty-one instances of the application all initially connect to QM1 (as it is the first queue manager defined in the ccdt ConfigMap).<\/p>\n<p>You can see this for yourself in the output from the <code class=\"mqunicode\">current-app-status.sh<\/code> script shown in the top-right of the video. If you look at the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/demo-scripts\/current-app-status.sh\">source for the script<\/a> you can see that it is just displaying output from <code class=\"mqunicode\">DIS APSTATUS<\/code>.<\/p>\n<p>The result looks like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/01-three-qmgrs-21-clients-unbalanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p>Without any manual intervention or reconfiguration, this is quickly rebalanced. Some of the client applications are automatically instructed to reconnect to QM2 and QM3, so that the client applications are evenly distributed amongst the three queue managers.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=112\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=114\">the demo video<\/a> (at approx 1min 52secs) the application instances are rebalanced so that there are seven instances connected to each queue manager.<\/p>\n<p>You can see this for yourself in the <code class=\"mqunicode\">DIS APSTATUS<\/code> output from the <code class=\"mqunicode\">current-app-status.sh<\/code> script shown in the top-right of the screen.<\/p>\n<p>The result is that the application instances are connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/02-three-qmgrs-21-clients-balanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<pre class=\"mqunistep\">.\/demo-scripts\/deploy-test-apps-randomly-distributed.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=208\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=208\">You can see this between 03:28 and 04:03 in the demo video<\/a><\/em><\/small><\/p>\n<p>Alternatively, you can choose to configure client apps to choose a queue manager in the cluster at random to initially connect to. The <code class=\"mqunicode\">deploy-test-apps-randomly-distributed.sh<\/code> script is an example of how to do this.<\/p>\n<p>This adds <code class=\"mqunicode\">connectionManagement<\/code> options to the CCDT file, specifying an affinity of <code class=\"mqunicode\">none<\/code> and giving each queue manager an equal client weight.<\/p>\n<p>The options look like this:<\/p>\n<pre class=\"mqunilisting\">\"connectionManagement\":\n{\n    \"clientWeight\": 1,\n    \"affinity\": \"none\"\n},<\/pre>\n<p>The <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/test-app-resources\/ccdt-randomly-distributed.yaml\">demo config for the CCDT file<\/a> shows how these options need to be added to each queue manager.<\/p>\n<p>If you choose to do this, as you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=224\">the demo video<\/a> (at approx 3min 44secs) all twenty-one instances of the application choose a queue manager at random to connect to do.<\/p>\n<p>The result in the demo video looked like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/01-three-qmgrs-21-clients-random.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p><em>Note that as each client made a random selection of queue manager, your distribution will be slightly different. But it should still be roughly even.<\/em><\/p>\n<p>In the same way as if you choose for client apps to make an initial connection to QM1, without any manual intervention or reconfiguration, the applications are quickly rebalanced and redistributed evenly amongst the three queue managers.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=241\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=241\">the demo video<\/a> (at approx 4min 1secs) the application instances are again rebalanced so that there are seven instances connected to each queue manager.<\/p>\n<p>You can see this for yourself in the <code class=\"mqunicode\">DIS APSTATUS<\/code> output from the <code class=\"mqunicode\">current-app-status.sh<\/code> script shown in the top-right of the screen.<\/p>\n<p>The result is that the application instances are connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/02-three-qmgrs-21-clients-balanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<h2 id=\"demo_scaleup_apps\">Load balancing when client apps are scaled up<\/h2>\n<pre class=\"mqunistep\">.\/demo-scripts\/scale-up-getter.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=292\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=292\">You can see this between 04:52 and 05:37 in the demo video<\/a><\/em><\/small><\/p>\n<p>The <code class=\"mqunicode\">scale-up-getter.sh<\/code> script scales up the getter application from twenty-one instances to thirty instances.<\/p>\n<p>You can see the list of the application instance pods in the output from the <code class=\"mqunicode\">show-getter-apps.sh<\/code> script in the bottom-right of the video (<a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/demo-scripts\/show-getter-apps.sh\">source<\/a>).<\/p>\n<p>As you can see from the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/demo-scripts\/scale-up-getter.sh\">source for the demo script<\/a>, this is simply done by scaling up the number of replicas for the application <code class=\"mqunicode\">Deployment<\/code>.<\/p>\n<pre class=\"mqunilisting\">oc scale deploy test-app-getter --replicas=30<\/pre>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=304\">the demo video<\/a> (at approx 5min 4secs) the new instances of the application all initially connect to QM1, as before. (To make this part of the demo clearer, the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/test-app-resources\/ccdt-start-with-qm1-first.yaml\">CCDT file was configured so that new application instances initially connect to QM1 because it is the first queue manager<\/a>, however as described above you can choose to configure the new instances to choose a queue manager at random instead.)<\/p>\n<p>The result is that the application instances are connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/03-three-qmgrs-30-clients-unbalanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p>As with the initial deployment, without any manual intervention or reconfiguration, the application is again quickly rebalanced. Some of the new application instances are automatically instructed to reconnect to QM2 and QM3, so that the client applications are again evenly distributed amongst the three queue managers.<\/p>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=332\">the demo video<\/a> (at approx 5min 32secs) the application instances are rebalanced so that there are now ten instances connected to each queue manager.<\/p>\n<p>You can see this for yourself in the <code class=\"mqunicode\">DIS APSTATUS<\/code> output from the <code class=\"mqunicode\">current-app-status.sh<\/code> script shown in the top-right of the video.<\/p>\n<p>The result is that the application instances are connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/04-three-qmgrs-30-clients-balanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<h2 id=\"demo_scaledown_apps\">Load balancing when client apps are scaled down<\/h2>\n<pre class=\"mqunistep\">.\/demo-scripts\/scale-getter-one-per-qmgr.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=178\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=338\">You can see this between 05:38 and 06:10 in the demo video<\/a><\/em><\/small><\/p>\n<p>The <code class=\"mqunicode\">scale-getter-one-per-qmgr.sh<\/code> script scales down the getter application to just three instances.<\/p>\n<p>As you can see from the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/demo-scripts\/scale-getter-one-per-qmgr.sh\">source for the demo script<\/a>, this is simply done by setting the number of replicas for the application <code class=\"mqunicode\">Deployment<\/code>.<\/p>\n<pre class=\"mqunilisting\">oc scale deploy test-app-getter --replicas=3<\/pre>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=357\">the demo video<\/a> (at approx 5min 57secs) because Kubernetes isn&#8217;t aware of where each application is connected, it initially removed all of the instances of the application connected to QM2, leaving two instances connected to QM1.<\/p>\n<p>The result is that the application instances were initially connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/05-three-qmgrs-3-clients-unbalanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p>As before, without any manual intervention or reconfiguration, the application is quickly rebalanced. One of the application instances connected to QM1 was automatically instructed to reconnect to QM2, so that the client applications are again evenly distributed amongst the three queue managers.<\/p>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=362\">the demo video<\/a> (at approx 6min 2secs) the application instances are rebalanced so that there is now one instance connected to each queue manager.<\/p>\n<p>The result is that the application instances are connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/06-three-qmgrs-3-clients-balanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<h2 id=\"demo_scaletoolow\">Load balancing if there aren&#8217;t enough client apps<\/h2>\n<pre class=\"mqunistep\">.\/demo-scripts\/scale-getter-one-only.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=373\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=373\">You can see this between 06:13 and 06:38 in the demo video<\/a><\/em><\/small><\/p>\n<p>The <code class=\"mqunicode\">scale-getter-one-only.sh<\/code> script scales the client application down so that only a single instance is running.<\/p>\n<p>As you can see from <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=382\">the demo video<\/a> (at approx 6min 22secs), the remaining instance of the application was connected to QM1. This happened by chance, and the application instance could have been connected to any of the queue managers.<\/p>\n<p>However, in this case, it was connected like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/07-three-qmgrs-1-client.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p>The problem with this is that there are now no applications consuming the messages from the queues on QM2 and QM3.<\/p>\n<p>As you can see in <a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=385\">the demo video<\/a> (at approx 6min 25secs) the number of messages on the queues on QM2 and QM3 continued to increase as a result.<\/p>\n<p>You can see this in the output from the <code class=\"mqunicode\">queue-depth.sh<\/code> script, shown in the middle of the right side of the screen. If you look at the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/demo-scripts\/queue-depth.sh\">source for the script<\/a> you can see that it is just displaying output from <code class=\"mqunicode\">DIS QLOCAL(APPQ1) CURDEPTH<\/code>.<\/p>\n<p>This highlights that it is important that the <code class=\"mqunicode\">replicas<\/code> for the <code class=\"mqunicode\">Deployment<\/code> of applications consuming from the cluster queue is set to <strong>at least<\/strong> the same value as the number of queue managers in your uniform cluster.<\/p>\n<h2>Scaling the application back up<\/h2>\n<pre class=\"mqunistep\">.\/demo-scripts\/scale-up-getter.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=310\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=442\">You can see this by 07:22 in the demo video<\/a><\/em><\/small><\/p>\n<p>Before proceeding to the next step in the demo, the number of instances of the client application is returned to thirty, so that there are again ten instances of the application connected to each of the three queue managers.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/04-three-qmgrs-30-clients-balanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<h2 class=\"mqunisection\" id=\"demo_scale_qmgrs\">Scaling the queue manager cluster<\/h2>\n<pre class=\"mqunistep\">.\/03-setup-additional-qmgr.sh<\/pre>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pc?t=452\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pc?t=452\">You can see this between 07:32 and 09:49 in the demo video<\/a><\/em><\/small><\/p>\n<p>The <code class=\"mqunicode\">02-setup-additional-qmgr.sh<\/code> script adds a fourth queue manager to the uniform cluster.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/08-four-qmgrs-30-clients-unbalanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n<p>This does a few things.<\/p>\n<p>It creates a new <code class=\"mqunicode\">ConfigMap<\/code> with an MQSC file that defines the addresses for the channels the new queue manager will use to join the cluster. The <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-resources-2\/configmap-qm4.yaml\">config for this demo<\/a> defines the cluster sender channel for both of the existing full repositories in the cluster, and the cluster receiver channel for the new queue manager.<\/p>\n<pre class=\"mqunilisting\">apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: mq-uniform-cluster-qm4-mqsc-cm\ndata:\n  qm4-config.mqsc: |-\n    alter chl(DEMOCLUSTER_QM1) chltype(CLUSSDR) conname('uniform-cluster-qm1-ibm-mq.uniform-cluster.svc(1414)')\n    alter chl(DEMOCLUSTER_QM2) chltype(CLUSSDR) conname('uniform-cluster-qm2-ibm-mq.uniform-cluster.svc(1414)')\n    alter chl(DEMOCLUSTER_QM4) chltype(CLUSRCVR) conname('uniform-cluster-qm4-ibm-mq.uniform-cluster.svc(1414)')<\/pre>\n<p>The new <code class=\"mqunicode\">QueueManager<\/code> spec uses this ConfigMap, together with the existing <code class=\"mqunicode\">ConfigMaps<\/code> created for the existing three queue managers. This can be seen in the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/mq-resources-2\/qmgr-qm4.yaml\">config for QM4<\/a> in this demo, which starts by pointing at the three config maps.<\/p>\n<pre class=\"mqunilisting\">apiVersion: mq.ibm.com\/v1beta1\nkind: QueueManager\nmetadata:\n  name: uniform-cluster-qm4\nspec:\n  queueManager:\n    name: QM4\n    ini:\n      - configMap:\n          items:\n          - config.ini\n          name: mq-uniform-cluster-ini-cm\n    mqsc:\n      - configMap:\n          items:\n          - common_config.mqsc\n          name: mq-uniform-cluster-mqsc-cm\n      - configMap:\n          name: mq-uniform-cluster-qm4-mqsc-cm\n          items:\n            - qm4-config.mqsc<\/pre>\n<p>As shown in the diagram above, initially the client applications remain connected to the existing three queue managers. The <code class=\"mqunicode\">02-setup-additional-qmgr.sh<\/code> script next updates the client applications to reflect the updated uniform cluster.<\/p>\n<p>First, the &#8220;ccdt&#8221; <code class=\"mqunicode\">ConfigMap<\/code> is updated to add the connection addresses for the new queue manager. As before, <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/test-app-resources-2\/ccdt-randomly-distributed.yaml\">two entries are added<\/a> for the new queue manager. The first is for the queue manager as a member of the cluster, with <code class=\"mqunicode\">queueManager<\/code> set to <code class=\"mqunicode\">DEMOCLUSTER<\/code>.<\/p>\n<pre class=\"mqunilisting\">{\n    \"name\": \"DEF.SVRCONN\",\n    \"clientConnection\": {\n        \"connection\": [\n            {\n                \"host\": \"uniform-cluster-qm4-ibm-mq.uniform-cluster\",\n                \"port\": 1414\n            }\n        ],\n        \"queueManager\": \"DEMOCLUSTER\"\n    },\n    \"connectionManagement\":\n    {\n      \"clientWeight\": 1,\n      \"affinity\": \"none\"\n    },\n    \"transmissionSecurity\": {\n        \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n    },\n    \"type\": \"clientConnection\"\n},<\/pre>\n<p>The second new entry is for the queue manager itself, with it&#8217;s own <code class=\"mqunicode\">queueManager<\/code> name.<\/p>\n<pre class=\"mqunilisting\">{\n    \"name\": \"DEF.SVRCONN\",\n    \"clientConnection\": {\n        \"connection\": [\n            {\n                \"host\": \"uniform-cluster-qm4-ibm-mq.uniform-cluster\",\n                \"port\": 1414\n            }\n        ],\n        \"queueManager\": \"QM4\"\n    },\n    \"connectionManagement\":\n    {\n      \"clientWeight\": 1,\n      \"affinity\": \"none\"\n    },\n    \"transmissionSecurity\": {\n        \"cipherSpecification\": \"ANY_TLS12_OR_HIGHER\"\n    },\n    \"type\": \"clientConnection\"\n},<\/pre>\n<p>Client applications don&#8217;t need to be restarted to pick up the updated CCDT ConfigMap. Modifying the ConfigMap causes the CCDT file in the JMS application pods to update, and the IBM MQ client library detects and responds to changes in CCDT files.<\/p>\n<p>When the CCDT config map is updated, the updated applications are quickly rebalanced across the four queue managers.<\/p>\n<p><iframe loading=\"lazy\" width=\"450\" height=\"253\" src=\"https:\/\/www.youtube.com\/embed\/-dVQhok61Pct=579\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><small><em><a href=\"https:\/\/youtu.be\/-dVQhok61Pct=579\">You can see this by 09:39 in the demo video<\/a><\/em><\/small><\/p>\n<p>You can see this for yourself in the output from the <code class=\"mqunicode\">current-app-status.sh<\/code> script shown in the top-right of the video. As before, the <a href=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/blob\/master\/demo-scripts\/current-app-status.sh\">script<\/a> is displaying output from <code class=\"mqunicode\">DIS APSTATUS<\/code>.<\/p>\n<p>The resulting connections looks like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/github.com\/dalelane\/mq-uniformclusters-openshift-tutorial\/raw\/master\/doc\/09-four-qmgrs-30-clients-balanced.png\" alt=\"diagram\" class=\"mqunidiagram\"\/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overview You&#8217;re running a cluster of IBM MQ queue managers in Red Hat OpenShift, together with a large number of client applications putting and getting messages to them. This workload will vary over time, so you need flexibility in how you scale all of this. This tutorial will show how you can easily scale the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[55],"class_list":["post-4639","post","type-post","status-publish","format-standard","hentry","category-code","tag-mq"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4639","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4639"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4639\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4639"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4639"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4639"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}