{"id":4409,"date":"2021-06-29T23:31:00","date_gmt":"2021-06-29T23:31:00","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4409"},"modified":"2021-06-30T00:56:10","modified_gmt":"2021-06-30T00:56:10","slug":"describing-kafka-security-in-asyncapi","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4409","title":{"rendered":"Describing Kafka security in AsyncAPI"},"content":{"rendered":"<p><strong>The new version of <a href=\"https:\/\/asyncapi.com\">AsyncAPI<\/a>, 2.1.0, was released today. One of the updates is that it lets you describe Kafka security mechanisms and protocols. In this post, I&#8217;ll show how you can do this, and how it relates to configuring a Kafka client.<\/strong><\/p>\n<style>table.asyncapitable {border-collapse: collapse;}table.asyncapitable, .asyncapitable th, .asyncapitable td {font-size: 9.5px;text-align: center;border: thin black solid;}.asyncapitable th {background-color: #f1f1f1;}.asyncapitable th, .asyncapitable td {padding: 3px;}<\/style>\n<div style=\"overflow-x:auto; margin: 15px 0 15px 0;\">\n<table class=\"asyncapitable\">\n<thead>\n<tr>\n<th colspan=\"2\">Kafka config<\/th>\n<th colspan=\"2\">means<\/th>\n<th colspan=\"2\">AsyncAPI<\/th>\n<\/tr>\n<tr>\n<th><a href=\"https:\/\/kafka.apache.org\/documentation\/#consumerconfigs_sasl.mechanism\" rel=\"nofollow\"><code>sasl.mechanism<\/code><\/a><\/th>\n<th><a href=\"https:\/\/kafka.apache.org\/documentation\/#consumerconfigs_security.protocol\" rel=\"nofollow\"><code>security.protocol<\/code><\/a><\/th>\n<th>encryption?<\/th>\n<th>auth?<\/th>\n<th>server <a href=\"https:\/\/www.asyncapi.com\/docs\/specifications\/2.0.0#fixed-fields-4\" rel=\"nofollow\"><code>protocol<\/code><\/a><\/th>\n<th>security scheme <a href=\"https:\/\/www.asyncapi.com\/docs\/specifications\/2.0.0#fixed-fields-21\" rel=\"nofollow\"><code>type<\/code><\/a><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td align=\"center\"><em>unset<\/em><\/td>\n<td><code>PLAINTEXT<\/code><\/td>\n<td align=\"center\">no<\/td>\n<td align=\"center\">no<\/td>\n<td><code>kafka<\/code><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>PLAIN<\/code><\/td>\n<td><code>SASL_PLAINTEXT<\/code><\/td>\n<td align=\"center\">no<\/td>\n<td align=\"center\">yes, using SASL\/PLAIN<\/td>\n<td><code>kafka<\/code><\/td>\n<td align=\"center\"><code>plain<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>SCRAM-SHA-256<\/code><\/td>\n<td><code>SASL_PLAINTEXT<\/code><\/td>\n<td align=\"center\">no<\/td>\n<td align=\"center\">yes, using SASL\/SCRAM<\/td>\n<td><code>kafka<\/code><\/td>\n<td align=\"center\"><code>scramSha256<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>SCRAM-SHA-512<\/code><\/td>\n<td><code>SASL_PLAINTEXT<\/code><\/td>\n<td align=\"center\">no<\/td>\n<td align=\"center\">yes, using SASL\/SCRAM<\/td>\n<td><code>kafka<\/code><\/td>\n<td align=\"center\"><code>scramSha512<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>OAUTHBEARER<\/code><\/td>\n<td><code>SASL_PLAINTEXT<\/code><\/td>\n<td align=\"center\">no<\/td>\n<td align=\"center\">yes, using OAuth<\/td>\n<td><code>kafka<\/code><\/td>\n<td align=\"center\"><code>oauth2<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>GSSAPI<\/code><\/td>\n<td><code>SASL_PLAINTEXT<\/code><\/td>\n<td align=\"center\">no<\/td>\n<td align=\"center\">yes, using GSSAPI<\/td>\n<td><code>kafka<\/code><\/td>\n<td align=\"center\"><code>gssapi<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><em>unset<\/em><\/td>\n<td><code>SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">no<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>PLAIN<\/code><\/td>\n<td><code>SASL_SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">yes, using SASL\/PLAIN<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><code>plain<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>SCRAM-SHA-256<\/code><\/td>\n<td><code>SASL_SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">yes, using SASL\/SCRAM<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><code>scramSha256<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>SCRAM-SHA-512<\/code><\/td>\n<td><code>SASL_SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">yes, using SASL\/SCRAM<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><code>scramSha512<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>OAUTHBEARER<\/code><\/td>\n<td><code>SASL_SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">yes, using OAuth<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><code>oauth2<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><code>GSSAPI<\/code><\/td>\n<td><code>SASL_SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">yes, using GSSAPI<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><code>gssapi<\/code><\/td>\n<\/tr>\n<tr>\n<td align=\"center\"><em>unset<\/em><\/td>\n<td><code>SSL<\/code><\/td>\n<td align=\"center\">yes<\/td>\n<td align=\"center\">yes, using mutual TLS<\/td>\n<td><code>kafka-secure<\/code><\/td>\n<td align=\"center\"><code>X509<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!--more-->What that table is obviously missing is some unnecessary interactivity!<\/p>\n<p>This means that if you need to configure your Kafka client with these properties:<\/p>\n<p><label for=\"asyncapi-saslmechanism\"><a href=\"https:\/\/kafka.apache.org\/documentation\/#consumerconfigs_sasl.mechanism\"><code>sasl.mechanism<\/code><\/a><\/label><code>=<\/code><select id=\"asyncapi-saslmechanism\" onchange=\"asyncapiupdate()\"><option value=\"\"><\/option><option value=\"PLAIN\">PLAIN<\/option><option value=\"SCRAM-SHA-256\">SCRAM-SHA-256<\/option><option value=\"SCRAM-SHA-512\">SCRAM-SHA-512<\/option><option value=\"OAUTHBEARER\">OAUTHBEARER<\/option><option value=\"GSSAPI\">GSSAPI<\/option><\/select><br \/>\n<label for=\"asyncapi-securityprotocol\"><a href=\"https:\/\/kafka.apache.org\/documentation\/#consumerconfigs_security.protocol\"><code>security.protocol<\/code><\/a><\/label><code>=<\/code><select id=\"asyncapi-securityprotocol\" onchange=\"asyncapiupdate()\"><option value=\"PLAINTEXT\">PLAINTEXT<\/option><option value=\"SASL_PLAINTEXT\">SASL_PLAINTEXT<\/option><option value=\"SSL\">SSL<\/option><option value=\"SASL_SSL\">SASL_SSL<\/option><\/select><\/p>\n<p>And would configure a Java Kafka client like this:<\/p>\n<pre class=\"asyncapi\"><span id=\"asyncapi-java\">Properties props = new Properties();\nprops.put(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, \n          \"mykafkacluster.org:8092\");\nprops.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG,\n          \"<span class=\"yamlem\" id=\"java-protocol\"><\/span>\");\n<span id=\"java-security\">props.put(SaslConfigs.SASL_MECHANISM,\n          \"<span class=\"yamlem\" id=\"java-mechanism\"><\/span>\");<\/span>\n...<\/span><\/pre>\n<p>Then you should describe this in an AsyncAPI document like this:<\/p>\n<pre class=\"asyncapi\"><span id=\"asyncapi-spec\"><span class=\"yamlkey\">asyncapi<\/span>: 2.1.0\n<span class=\"yamlkey\">info<\/span>:\n  <span class=\"yamlkey\">title<\/span>: My Kafka topic\n  <span class=\"yamlkey\">version<\/span>: 1.0.0\n<span class=\"yamlkey\">servers<\/span>:\n  demo:\n    <span class=\"yamlkey\">url<\/span>: mykafkacluster.org:8092\n    <span class=\"yamlkey\">protocol<\/span>: <span class=\"yamlem\" id=\"asyncapi-protocol\"><\/span>\n    <span id=\"asyncapi-security\"><span class=\"yamlkey\">security<\/span>:\n      - mySecurity: []<\/span>\n<span id=\"asyncapi-components\"><span class=\"yamlkey\">components<\/span>:\n  <span class=\"yamlkey\">securitySchemes<\/span>:\n    mySecurity:\n      type: <span class=\"yamlem\" id=\"asyncapi-securityschemetype\"><\/span><\/span>\n<span class=\"yamlkey\">channels<\/span>:\n  MY.TOPIC:\n    <span class=\"yamlkey\">subscribe<\/span>:\n      <span class=\"yamlkey\">message<\/span>:\n        <span class=\"yamlkey\">name<\/span>: myMessage\n        <span class=\"yamlkey\">payload<\/span>:\n          <span class=\"yamlkey\">type<\/span>: object\n          <span class=\"yamlkey\">properties<\/span>:\n            val:\n              <span class=\"yamlkey\">type<\/span>: string<\/span>\n<\/pre>\n<p><script>function asyncapiupdatespec(e,a){document.getElementById(\"asyncapi-spec\").classList.remove(\"hide\"),document.getElementById(\"asyncapi-java\").classList.remove(\"hide\"),document.getElementById(\"asyncapi-protocol\").innerHTML=e,document.getElementById(\"asyncapi-securityschemetype\").innerHTML=a,a?(document.getElementById(\"asyncapi-security\").classList.remove(\"hide\"),document.getElementById(\"asyncapi-components\").classList.remove(\"hide\")):(document.getElementById(\"asyncapi-security\").classList.add(\"hide\"),document.getElementById(\"asyncapi-components\").classList.add(\"hide\"))}function asyncapiupdate(){var e=document.getElementById(\"asyncapi-saslmechanism\").value,a=document.getElementById(\"asyncapi-securityprotocol\").value;document.getElementById(\"java-protocol\").innerHTML=a,document.getElementById(\"java-mechanism\").innerHTML=e,e?document.getElementById(\"java-security\").classList.remove(\"hide\"):document.getElementById(\"java-security\").classList.add(\"hide\"),\"\"===e&&\"PLAINTEXT\"===a?asyncapiupdatespec(\"kafka\",\"\"):\"PLAIN\"===e&&\"SASL_PLAINTEXT\"===a?asyncapiupdatespec(\"kafka\",\"plain\"):\"SCRAM-SHA-256\"===e&&\"SASL_PLAINTEXT\"===a?asyncapiupdatespec(\"kafka\",\"scramSha256\"):\"SCRAM-SHA-512\"===e&&\"SASL_PLAINTEXT\"===a?asyncapiupdatespec(\"kafka\",\"scramSha512\"):\"OAUTHBEARER\"===e&&\"SASL_PLAINTEXT\"===a?asyncapiupdatespec(\"kafka\",\"oauth2\"):\"GSSAPI\"===e&&\"SASL_PLAINTEXT\"===a?asyncapiupdatespec(\"kafka\",\"gssapi\"):\"\"===e&&\"SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"\"):\"PLAIN\"===e&&\"SASL_SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"plain\"):\"SCRAM-SHA-256\"===e&&\"SASL_SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"scramSha256\"):\"SCRAM-SHA-512\"===e&&\"SASL_SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"scramSha512\"):\"OAUTHBEARER\"===e&&\"SASL_SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"oauth2\"):\"GSSAPI\"===e&&\"SASL_SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"gssapi\"):\"\"===e&&\"SSL\"===a?asyncapiupdatespec(\"kafka-secure\",\"X509\"):(document.getElementById(\"asyncapi-spec\").classList.add(\"hide\"),document.getElementById(\"asyncapi-java\").classList.add(\"hide\"))}asyncapiupdate();<\/script><\/p>\n<p>For more info about what is new in AsyncAPI 2.1.0, check out the <a href=\"https:\/\/www.asyncapi.com\/blog\/release-notes-2.1.0\">release notes<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The new version of AsyncAPI, 2.1.0, was released today. One of the updates is that it lets you describe Kafka security mechanisms and protocols. In this post, I&#8217;ll show how you can do this, and how it relates to configuring a Kafka client. Kafka config means AsyncAPI sasl.mechanism security.protocol encryption? auth? server protocol security scheme [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,602,584],"class_list":["post-4409","post","type-post","status-publish","format-standard","hentry","category-code","tag-apachekafka","tag-asyncapi","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4409","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4409"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4409\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4409"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4409"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4409"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}