IBM App Connect Enterprise (ACE) is a broker for developing and hosting high-throughput, high-scale integrations between a large number of applications and systems, including Apache Kafka.
In this post, I’ll describe how to use App Connect Enterprise to process Kafka messages that were serialized to a stream of bytes using Apache Avro schemas.

Background
Best practice when using Apache Kafka is to define Apache Avro schemas with a definition of the structure of your Kafka messages.
(For more detail about this, see my last post on From bytes to objects: describing Kafka events, or the intro to Avro that I wrote a couple of years ago.)
In this post, I’m assuming that you have embraced Avro, and you have Kafka topics with messages that were serialized using Avro schemas.
Perhaps you used a Java producer with an Avro SerDe that handled the serialization automatically for you.
Or your messages are coming from a Kafka Connect source connector, with an Avro converter that is handling the serialization for you.
Or you are doing the serialization yourself, such as if you’re producing Avro-serialized messages from a Python app.
Now you want to use IBM App Connect Enterprise to develop and host integrations for processing those Kafka messages. But you need App Connect to know how to:
- retrieve the Avro schemas it needs
- use the schemas to turn the binary stream of bytes on your Kafka topics into structured objects that are easy for ACE to manipulate and process
Read the rest of this entry »