{"id":5072,"date":"2024-03-13T20:58:05","date_gmt":"2024-03-13T20:58:05","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5072"},"modified":"2026-04-02T17:37:44","modified_gmt":"2026-04-02T17:37:44","slug":"getting-demo-events-onto-ibm-event-streams-topics-in-a-hurry","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5072","title":{"rendered":"Getting demo events onto IBM Event Streams topics in a hurry"},"content":{"rendered":"<p><strong>Sharing a couple of tips for quick-and-dirty demo setups.<\/strong><\/p>\n<p>I often need to put together <a href=\"https:\/\/dalelane.co.uk\/blog\/?p=4905\">demos of IBM Event Automation<\/a> without much notice. The starting point is almost always needing to get a bunch of interesting events onto a Kafka topic. <\/p>\n<p>What I need is a jumping-off point to illustrate the benefit of sharing streams of events in <strong>Event Endpoint Management<\/strong>, or the types of processing you can do in <strong>Event Processing<\/strong>. And to do that, I need a topic with events on them that will look interesting or relevant to who I&#8217;m demo&#8217;ing to. <\/p>\n<p>If I&#8217;ve got time to do this properly, I&#8217;ll setup a generator that will give me a continuous stream of randonly-generated events (<a href=\"https:\/\/github.com\/IBM\/kafka-connect-loosehangerjeans-source\">example<\/a>). But if I&#8217;m in a hurry, I&#8217;ll use the <a href=\"https:\/\/ibm.github.io\/event-automation\/es\/connecting\/rest-api\/\">REST Producer API<\/a> and do something like this instead.<\/p>\n<p><!--more--><\/p>\n<h3>Step 1 : <br \/>Get the URL for the REST Producer API<\/h3>\n<p>You can find this in the status section of the Event Streams custom resource. Or you can find it in the Event Streams admin UI. <\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-01.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-01.png\"\/><\/a><\/p>\n<h3>Step 2 : <br \/>Get credentials to use with the REST Producer API<\/h3>\n<p>You can do this by creating a KafkaUser custom resource. Or you can click on the <strong>Generate credentials<\/strong> button in the Event Streams UI next to the Producer endpoint URL. <\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-02.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-02.png\"\/><\/a><\/p>\n<p>Choose <strong>SCRAM username and password<\/strong>.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-03.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-03.png\"\/><\/a><\/p>\n<p>Choose <strong>Produce messages<\/strong>.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-04.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-04.png\"\/><\/a><\/p>\n<p>Name the topic you will be producing demo messages to.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-05.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-05.png\"\/><\/a><\/p>\n<p>Copy the username and password that is displayed. <\/p>\n<h3>Step 3 : <br \/>Get some test data &#8211; one event per line<\/h3>\n<p>There are loads of sites that will generate random data. <\/p>\n<p><a href=\"https:\/\/www.mockaroo.com\"><strong>Mockaroo<\/strong><\/a> is a good example &#8211; it generates random values for fields with different types, and will let you download all of that to a file.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-06.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-06.png\"\/><\/a><\/p>\n<p>What you need to do will depend on which site you pick, but in general you choose a few fields, decide what names to give them, and get the site to generate random values for them.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-07.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-07.png\"\/><\/a><\/p>\n<p>In this example, I&#8217;m generating JSON data, and by unticking the array checkbox I get <a href=\"https:\/\/jsonlines.org\">jsonlines<\/a> data (that is, one file with lots of JSON objects, one JSON object per line).<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-08.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-08.png\"\/><\/a><\/p>\n<h3>Step 4 : <br \/>Run this shell script<\/h3>\n<p>It just needs to read the file a line at a time, using <code>curl<\/code> to POST each line to the Producer API.<\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/390176f3a1a4b29abd2b908e4b5c514e.js\"><\/script><small><a href=\"https:\/\/gist.github.com\/dalelane\/390176f3a1a4b29abd2b908e4b5c514e\"><code>send-file.sh<\/code> on github<\/a><\/small><\/p>\n<h3>Step 5 : <br \/>Review the events<\/h3>\n<p>Look at your topic in the Event Streams admin UI. You&#8217;ve now got a topic with a batch of events on, ready to use for a demo.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-09.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-09.png\"\/><\/a><\/p>\n<h3>Variation : <br \/>Test data &#8211; one event per file<\/h3>\n<p>Some data generator sites don&#8217;t give you a single file, but generate a separate file for each JSON object.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-10.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-10.png\"\/><\/a><\/p>\n<p>In that case, the shell script is slightly different. Put all of the files in a folder, and use curl to POST each file in the folder to the Producer API.<\/p>\n<p><script src=\"https:\/\/gist.github.com\/dalelane\/d4d7c5d7689322287ba7d53a3a148b39.js\"><\/script><small><a href=\"https:\/\/gist.github.com\/dalelane\/d4d7c5d7689322287ba7d53a3a148b39\"><code>send-folder<\/code> on github<\/a><\/small><\/p>\n<p>The result is the same &#8211; a topic with events ready for a demo.<\/p>\n<p><a href=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-11.png\"><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-03-13-es-rest-api\/screenshot-11.png\"\/><\/a><\/p>\n<h4>That&#8217;s it.<\/h4>\n<p>Super simple, but putting this here will hopefully save people a little time in getting their demo ready. <\/p>\n<p>It&#8217;s not a very elegant way of putting data on a Kafka topic. It&#8217;s certainly a lot less efficient than using a proper Kafka client. But if you&#8217;re in a hurry, it works and it doesn&#8217;t need anything other than curl to work.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sharing a couple of tips for quick-and-dirty demo setups. I often need to put together demos of IBM Event Automation without much notice. The starting point is almost always needing to get a bunch of interesting events onto a Kafka topic. What I need is a jumping-off point to illustrate the benefit of sharing streams [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,583,584],"class_list":["post-5072","post","type-post","status-publish","format-standard","hentry","category-code","tag-apachekafka","tag-ibmeventstreams","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5072","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5072"}],"version-history":[{"count":1,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5072\/revisions"}],"predecessor-version":[{"id":5961,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5072\/revisions\/5961"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}