{"id":4219,"date":"2020-11-27T12:55:37","date_gmt":"2020-11-27T12:55:37","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4219"},"modified":"2020-11-27T13:02:08","modified_gmt":"2020-11-27T13:02:08","slug":"describing-kafka-with-asyncapi","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4219","title":{"rendered":"Describing Kafka with AsyncAPI"},"content":{"rendered":"<p><strong>In this post, I want to describe how to use AsyncAPI to document how you&#8217;re using Apache Kafka. There are already great AsyncAPI <a href=\"https:\/\/www.asyncapi.com\/docs\/getting-started\">&#8220;Getting Started&#8221;<\/a> guides, but it supports a variety of protocols, and I haven&#8217;t found an introduction written specifically from the perspective of a Kafka user.<\/strong><\/p>\n<p>I&#8217;ll start with a description of what AsyncAPI is.<\/p>\n<blockquote style=\"font-family: Georgia, Times, Times New Roman\"><p>&#8220;an open source initiative &#8230; goal is to make working with Event-Driven Architectures as easy as it is to work with REST APIs &#8230; from documentation to code generation, from discovery to event management&#8221;<\/p>\n<p><small><a href=\"https:\/\/asyncapi.com\/docs\" target=\"_blank\" rel=\"noopener noreferrer\">asyncapi.com\/docs<\/a><\/small><\/p><\/blockquote>\n<p>The most obvious initial aspect is that it is a way to document how you&#8217;re using Kafka topics, but the impact is broader than that: a consistent approach to documentation enables an ecosystem that includes things like automated code generation and discovery.<\/p>\n<p><!--more--><br \/>\n<a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-3.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-3.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<table style=\"min-width: 448px; border-collapse: collapse; border: thin black solid; text-align: center;\" border=\"1\">\n<tbody>\n<tr>\n<th>what Kafka calls<\/th>\n<th>AsyncAPI describes as<\/th>\n<\/tr>\n<tr>\n<td>broker<\/td>\n<td>message broker<\/td>\n<\/tr>\n<tr>\n<td>producer<\/td>\n<td>publisher<\/td>\n<\/tr>\n<tr>\n<td>consumer<\/td>\n<td>subscriber<\/td>\n<\/tr>\n<tr>\n<td>event \/ record \/ message<\/td>\n<td>message<\/td>\n<\/tr>\n<tr>\n<td>topic<\/td>\n<td>channel<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Most terminology in AsyncAPI will be recognizable to Kafka users. Describing Kafka Producers as &#8220;publishers&#8221; and Kafka Consumers as &#8220;subscribers&#8221; isn&#8217;t that unusual, and something I&#8217;ve heard people do before, particularly for users who&#8217;ve come from messaging worlds like JMS.<\/p>\n<p>Perhaps more unexpected is that, in AsyncAPI, Kafka topics are described as &#8220;channels&#8221;.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-5.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-5.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist-it.appspot.com\/https:\/\/github.com\/project-flogo\/asyncapi\/raw\/master\/examples\/kafka\/asyncapi_secure.yml?footer=minimal\"><\/script><\/p>\n<p>AsyncAPI specs are a YAML file (or you can use JSON if you prefer), that formally documents how to connect to the Kafka cluster, the details of the Kafka topic(s), and the type of data in the messages on the topics. It includes both formal schema definitions and space for free-text descriptions.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-6.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-6.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>This graphic is used on the AsyncAPI web site to describe the structure of an AsyncAPI spec.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-7.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-7.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>It&#8217;s a bit taller than I&#8217;m showing here, but these are the important sections.<\/p>\n<p>The bottom bits in &#8220;Components&#8221; are where you can put definitions that can be reused by reference throughout the rest of your spec.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-8.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-8.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>For this post, I want to go through the different sections in this structure, and describe what they mean through the lens of documenting Kafka.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-9.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-9.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>I&#8217;ll start with the simplest bit: uniquely identifying your spec.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-10.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-10.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/d684c0eefbf5c5dae871391c6c34a39e.js\"><\/script><\/p>\n<p>AsyncAPI recommends using URNs for this, although I have seen some examples that use URLs.<\/p>\n<p>The important thing is that you have an id with a unique string.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-11.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-11.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Next, you can provide high-level info about what you&#8217;re documenting in this spec.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-12.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-12.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Aside from the obvious fields like a title, version number and contact details, the interesting bit here is a description. This is a place to capture some context for the use of Kafka being defined. And you can use markdown to include rich-text formatting.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-13.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-13.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist-it.appspot.com\/https:\/\/github.com\/asyncapi\/asyncapi\/raw\/94baeb8165e64ccd0c364801c28d473770983f86\/versions\/2.0.0\/asyncapi.md?footer=minimal&amp;slice=234:244\"><\/script><\/p>\n<p>The result can look something like this.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-14.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-14.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Next, you provide details of your Kafka cluster, so people know how to connect their client applications.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-15.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-15.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/25812fd8403a5c470b398f636470d366.js\"><\/script><\/p>\n<p>The servers section of the spec is made up of a list of <code>server<\/code> objects, each defining a Kafka broker and each identified by a unique name.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-16.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-16.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>The most important bit is the URL field, where you provide the connection address for the Kafka broker.<\/p>\n<p>The other bit you need to specify is the protocol. AsyncAPI can be used to document a variety of systems, so here is where you identify that you&#8217;re using Kafka.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-17.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-17.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Notice that there are two flavours of the protocol for Kafka.<\/p>\n<p>If your Kafka cluster doesn&#8217;t have auth enabled, then you use the protocol <code>kafka<\/code>.<\/p>\n<p>If client applications are required to provide credentials, then you identify this by using the protocol <code>kafka-secure<\/code>.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-18.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-18.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/f2d4805d99bc286f196a768ebe883ee8.js\"><\/script><\/p>\n<p>The result looks something like this.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-19.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-19.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/e19b6d3df303f70b17cd1cca7527dc5f.js\"><\/script><\/p>\n<p>As you&#8217;ll have more than one Kafka broker in your cluster, you&#8217;ll probably want to identify each of them.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-20.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-20.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist-it.appspot.com\/https:\/\/github.com\/asyncapi\/java-spring-template\/raw\/master\/template\/src\/main\/resources\/application.yml?footer=minimal&amp;slice=59:65\"><\/script><\/p>\n<p>This means that the bootstrap address for Kafka clients wanting to connect should be formed by combining the URLs for each of the server objects. (And this is what AsyncAPI&#8217;s <a href=\"https:\/\/github.com\/asyncapi\/java-spring-template\/blob\/80832faa836b9f9dc3022f15d8f57a8f206416e3\/template\/src\/main\/resources\/application.yml#L61-L64\">Java Spring code generator<\/a> does.)<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-21.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-21.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>This is a limitation, as it prevents you from including multiple Kafka clusters (such as a production cluster and dev\/test\/staging clusters) in a single AsyncAPI spec. I think it would help to extend the spec to enable identifying multiple clusters, which is <a href=\"https:\/\/github.com\/asyncapi\/asyncapi\/issues\/465\">a suggestion I&#8217;ve raised with the AsyncAPI community<\/a>.<\/p>\n<p>In the meantime, you could just list all brokers from all your clusters, and rely on using the description or extension fields to explain which ones are in which cluster. (You would have to be careful of code generators or other parts of the AsyncAPI ecosystem that will misinterpret them as all being members of one large cluster.)<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-23.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-23.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/b6904f3cad1f72e549e0dd9d7aab1316.js\"><\/script><\/p>\n<p>For those of us who are running Kafka in Kubernetes, and fronting it with a single bootstrap service or route that round-robins each broker in the cluster, we can just use that address in a single server object.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-24.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-24.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/0372c27fbe900f6f1a22421dc6d3c3c7.js\"><\/script><\/p>\n<p>The same limitation about multiple clusters will apply though.<\/p>\n<p>You could identify multiple Kafka clusters, each as a separate server object, and rely on naming or description fields to make it clear that these are actually separate clusters. But this wouldn&#8217;t be consistent with some existing use of AsyncAPI, or the assumptions in supporting tools in the ecosystem, like the Java Spring code generator.<\/p>\n<p>For now, I think better to avoid doing this.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-25.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-25.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/92db422912340c0ca4e36085e864757c.js\"><\/script><\/p>\n<p>As I mentioned above, if your Kafka cluster is secured, you identify this by specifying <code>kafka-secure<\/code> as the protocol.<\/p>\n<p>You identify the type of credentials by adding a <code>security<\/code> section to the <code>server<\/code> object. The value you put in there is the name of a <code>securityScheme<\/code> object you define in the <code>components<\/code> section.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-26.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-26.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Notice that the contents of the <code>security<\/code> value in the <code>server<\/code> object is just <code>[]<\/code>. The important bit is the name, which matches up with the details down in <code>components.securitySchemes<\/code>.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-27.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-27.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>The types of security scheme that you can specify aren&#8217;t Kafka-specific, so the best option is to choose the value that describes your type of approach to security.<\/p>\n<p>For example, if you&#8217;re using SASL\/SCRAM, that is a username\/password-based approach to auth, so you could describe this as <code>userPassword<\/code>.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-28.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-28.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/89f3d54d9b8bcf163cd283f315d59739.js\"><\/script><\/p>\n<p>If you want to be more specific about the security options that Kafka clients need to use, then you could explain that in the description field, or you could use extensions to document it.<\/p>\n<p>As with OpenAPI, you can add additional attributes to the spec by prefixing them with <code>x-<\/code> to identify them as your own extensions to AsyncAPI.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-29.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-29.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>The problem with either of these approaches is that it won&#8217;t lead to people documenting these standard aspects of configuring Kafka clients in a consistent way, and would be harder to exploit in things like code generators.<\/p>\n<p>I think it would help to extend the spec to include Kafka-specific security config options, which is <a href=\"https:\/\/github.com\/asyncapi\/asyncapi\/issues\/466\">a suggestion I&#8217;ve raised with the AsyncAPI community<\/a>.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-31.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-31.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>The next thing to do is to identify your Kafka topics.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-32.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-32.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/8077e7c8a3210b19d6d34a7db29fda2a.js\"><\/script><\/p>\n<p>As I mentioned above, in AsyncAPI you describe your topics as channels.<\/p>\n<p>The <code>channels<\/code> section is made up of channel objects, each named using the name of your topic.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-33.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-33.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>For each topic, you need to identify the operations that you want to describe in the spec.<\/p>\n<p>As I mentioned above, AsyncAPI describes producing and consuming as publish and subscribe operations.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-34.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-34.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/ae9d85a5ce3a12524f3ce8d7102d44b7.js\"><\/script><\/p>\n<p>You can start by describing the operation &#8211; giving it a unique id, a short one-line text summary, and a more detailed description (which can include markdown formatting).<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-35.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-35.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/e5957f788f0f883e8bd3ff3f5e95ce58.js\"><\/script><\/p>\n<p>AsyncAPI puts protocol-specific values in sections called <code>bindings<\/code>.<\/p>\n<p>Next, you can specify the values that Kafka clients should use to perform this operation in a bindings section.<\/p>\n<p>The values you can describe are the consumer group id, and the client id.<\/p>\n<p>If there are expectations about the format of these values, then you can describe them here, such as by using regular expressions.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-36.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-36.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/4d45b79e196b0d3ca1f9b95cae9d32fa.js\"><\/script><\/p>\n<p>Alternatively, if there is a discrete set of valid values, then you can enumerate all of them here instead.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-37.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-37.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Note that these are the only Kafka-specific attributes that are included in the bindings for Kafka operations.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-38.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-38.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Next, you describe the messages on the topic.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-39.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-39.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/ce78abbb68fe011e3d787047d97e64ba.js\"><\/script><\/p>\n<p>As with all the other levels of the spec, you can provide background and narrative in a description field.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-40.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-40.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/25b959347bd772f3be68d8f7094a91f3.js\"><\/script><\/p>\n<p>Again, Kafka-specific values go into a <code>bindings<\/code> section. For messages, the value you can describe is how keys are used in messages on this topic.<\/p>\n<p>You can describe the type &#8211; such as identifying whether you are using numeric or string-based keys. You can provide a regex if there is a pattern to how keys are defined. Or if you&#8217;re using a predefined, discrete set of keys, you can list them all in an enum.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-41.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-41.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>As before, note that this is the only Kafka-specific attribute that is included in the bindings for Kafka messages.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-42.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-42.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Next, you document the headers on the messages.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-43.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-43.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/9a692d3e087b3d2dcd0a97432bb89edd.js\"><\/script><\/p>\n<p>You can list each header as a property of the headers object, and for each header provide a description for what it is for, and the type of data in the value.<\/p>\n<p>There is also space to include a set of examples of what the headers can look like.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-44.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-44.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>Finally, you describe the message body.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-45.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-45.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/7d753d01ce185f209eb9dfb583c43f13.js\"><\/script><\/p>\n<p>By default, you do this using AsyncAPI&#8217;s own schema format.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-46.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-46.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/87aec9fde29ff3f684d06d038205b3fd.js\"><\/script><\/p>\n<p>This means identifying the type of data, any restrictions that apply, and providing some examples.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-47.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-47.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/9bbf404dec5e711a1317cdef88116bc4.js\"><\/script><\/p>\n<p>If messages contain multiple fields, you can identify all of these, and specify which ones are required and which are optional.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-48.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-48.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>But you don&#8217;t have to use AsyncAPI&#8217;s own schema format &#8211; a few other approaches are supported.<\/p>\n<p>For Kafka users, the <a href=\"https:\/\/dalelane.co.uk\/blog\/?tag=apacheavro\">most useful of these is likely to be Apache Avro<\/a>. If you&#8217;re using Avro to serialize and deserialize your messages (and you really should), then you can include a reference to your Avro schema.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-49.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-49.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/7f14a1bc2103f0c93b53d68b6aa9f532.js\"><\/script><\/p>\n<p>For example, if your AsyncAPI spec is in a file on a filesystem, you can provide the relative location of the Avro schema file.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-50.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-50.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/6864fe8c267dcc8225971a9c6065a758.js\"><\/script><\/p>\n<p>Alternatively, you can provide an absolute URL for where the schema is hosted, such as in a schema registry.<\/p>\n<p>Reusing your existing Avro schemas means you don&#8217;t need to define the data types for your Kafka payloads multiple times, and the AsyncAPI spec supplements what Avro captures with information about the topics and Kafka clusters.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-51.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-51.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/adbe58e7b07ab67525b6740dcfef7a60.js\"><\/script><\/p>\n<p>And that&#8217;s it.<\/p>\n<p>I think there is a lot of value in capturing this detail about how you&#8217;re using Kafka in a consistent way.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-52.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-52.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>I&#8217;ve tried playing with a couple of examples of supporting tools in the ecosystem.<\/p>\n<p>I&#8217;ve already mentioned the <a href=\"https:\/\/github.com\/asyncapi\/java-spring-template\/\">Java Spring code generation<\/a> &#8211; point it at an AsyncAPI spec, and it generates the source code for a Java app ready to start running against your Kafka cluster, with the connection information already set up.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-53.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-53.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p><a href=\"http:\/\/microcks.io\/\">Microcks<\/a> is another interesting tool. It deploys it&#8217;s own Kafka broker, and if you upload your AsyncAPI spec it creates Kafka topics based on the description in the spec, and starts generating and producing mock data to them on a frequency interval that you specify.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-54.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-54.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>It means you can define and document the data you plan to produce to your Kafka topics, and let Microcks set up a topic with a live stream of mock data matching that spec. And that lets you start developing your Kafka consumers against the mock topic without needing to wait until you have a real topic with real data.<\/p>\n<p><a href=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-55.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"slideimg201127\" src=\"https:\/\/dalelane.co.uk\/blog\/post-images\/201127-asyncapi\/201127-asyncapi-sm-55.png\" alt=\"screenshot - click to enlarge\" title=\"screenshot - click to enlarge\"\/><\/a><\/p>\n<p>I&#8217;m sure there is more that we can do with this, but this is an intro to starting to use AsyncAPI to describe your Kafka topics.<\/p>\n<p>For more information, look at <a href=\"https:\/\/asyncapi.com\/\">asyncapi.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this post, I want to describe how to use AsyncAPI to document how you&#8217;re using Apache Kafka. There are already great AsyncAPI &#8220;Getting Started&#8221; guides, but it supports a variety of protocols, and I haven&#8217;t found an introduction written specifically from the perspective of a Kafka user. I&#8217;ll start with a description of what [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[595,593,602,594,584],"class_list":["post-4219","post","type-post","status-publish","format-standard","hentry","category-code","tag-apacheavro","tag-apachekafka","tag-asyncapi","tag-avro","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4219","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4219"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4219\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4219"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4219"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4219"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}