{"id":4518,"date":"2021-10-25T21:41:31","date_gmt":"2021-10-25T21:41:31","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4518"},"modified":"2024-05-13T22:45:01","modified_gmt":"2024-05-13T22:45:01","slug":"processing-apache-avro-serialized-kafka-messages-with-ibm-app-connect-enterprise","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4518","title":{"rendered":"Processing Apache Avro-serialized Kafka messages with IBM App Connect Enterprise"},"content":{"rendered":"

UPDATE: Please see the updated version of this post<\/a><\/em><\/strong><\/p>\n

IBM App Connect Enterprise<\/a> (ACE) is a broker for developing and hosting high-throughput, high-scale integrations between a large number of applications and systems, including Apache Kafka.<\/strong><\/p>\n

In this post, I’ll describe how to use App Connect Enterprise to process Kafka messages that were serialized to a stream of bytes using Apache Avro schemas.<\/strong><\/p>\n

\"screenshot\"<\/p>\n

Background<\/h3>\n

Best practice when using Apache Kafka<\/a> is to define Apache Avro<\/a> schemas with a definition of the structure of your Kafka messages.<\/p>\n

(For more detail about this, see my last post on From bytes to objects: describing Kafka events<\/a>, or the intro to Avro<\/a> that I wrote a couple of years ago.)<\/p>\n

In this post, I’m assuming that you have embraced Avro, and you have Kafka topics with messages that were serialized using Avro schemas.<\/p>\n

Perhaps you used a Java producer with an Avro SerDe<\/a> that handled the serialization automatically for you.<\/p>\n

Or your messages are coming from a Kafka Connect source connector, with an Avro converter<\/a> that is handling the serialization for you.<\/p>\n

Or you are doing the serialization yourself, such as if you’re producing Avro-serialized messages from a Python app<\/a>.<\/p>\n

Now you want to use IBM App Connect Enterprise<\/a> to develop and host integrations for processing those Kafka messages. But you need App Connect to know how to:<\/p>\n