{"id":3892,"date":"2019-10-17T16:04:37","date_gmt":"2019-10-17T16:04:37","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=3892"},"modified":"2019-10-17T16:04:37","modified_gmt":"2019-10-17T16:04:37","slug":"using-avro-schemas-from-python-apps-with-ibm-event-streams","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=3892","title":{"rendered":"Using Avro schemas from Python apps with IBM Event Streams"},"content":{"rendered":"<p><strong>I&#8217;ve written before about <a href=\"https:\/\/dalelane.co.uk\/blog\/?p=3781\">how to write a schema<\/a> for your developers using Kafka. The examples I used before were all in Java, but someone asked me yesterday if I could share some Python equivalents.<\/strong><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/dalelane.co.uk\/blog\/wp-content\/uploads\/2019\/10\/schema.png\" style=\"border: thin black solid\"\/><\/p>\n<p>The principles are described in the <a href=\"https:\/\/ibm.github.io\/event-streams\/schemas\/setting-nonjava-apps\/\">Event Streams documentation<\/a>, but in short, your Kafka producers use <a href=\"https:\/\/avro.apache.org\">Apache Avro<\/a> to serialize the message data that you send, and identify the schema that you&#8217;ve used in the Kafka message header. In your Kafka consumers, you look at the headers of the messages that you receive to know which schema to retrieve, and use that to deserialize message data.<\/p>\n<p><!--more-->The interesting bit was how to do Avro serialization \/ deserialization in Python:<\/p>\n<pre style=\"border: thin solid silver; background-color: #eeeeee; padding: 0.7em\">import io\nfrom avro.schema import Parse\nfrom avro.io import DatumWriter, DatumReader, BinaryEncoder, BinaryDecoder\n\n#\n# Get the binary encoding for the provided object, using the\n# provided Avro schema definition.\n#\n# @param myschema - Avro schema object\n# @param myobject - Python object to be serialized\n#\n# @returns bytes representation of the binary-encoded serialized object\ndef serialize(myschema, myobject):\n  buf = io.BytesIO()\n  encoder = BinaryEncoder(buf)\n  writer = DatumWriter(writer_schema=myschema)\n  writer.write(myobject, encoder)\n  buf.seek(0)\n  return (buf.read())\n\n#\n# Extracts a Python object from a binary-encoded serialization\n# using the provided Avro schema definition.\n#\n# @param myschema - Avro schema object\n# @param mydata   - bytes to be deserialized\n#\n# @returns Python object deserialized from the provided bytes\ndef deserialize(myschema, mydata):\n  buf = io.BytesIO(mydata)\n  decoder = BinaryDecoder(buf)\n  reader = DatumReader(writer_schema=myschema)\n  return (reader.read(decoder))<\/pre>\n<p>I&#8217;ve shared <a href=\"https:\/\/github.com\/dalelane\/event-streams-schemas-python\">a working sample consumer and producer app on Github<\/a> so you can see how this all fits in in the context of a Python Kafka app. The <code><a href=\"https:\/\/github.com\/dalelane\/event-streams-schemas-python\/blob\/master\/README.md\">README.md<\/a><\/code> tells you how to configure it to point to your Event Streams cluster, but other than that it&#8217;s ready to run as-is.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve written before about how to write a schema for your developers using Kafka. The examples I used before were all in Java, but someone asked me yesterday if I could share some Python equivalents. The principles are described in the Event Streams documentation, but in short, your Kafka producers use Apache Avro to serialize [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":3893,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[595,593,594,582,583,584],"class_list":["post-3892","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-apacheavro","tag-apachekafka","tag-avro","tag-eventstreams","tag-ibmeventstreams","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3892","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3892"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3892\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/3893"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3892"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3892"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3892"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}