{"id":5298,"date":"2024-10-10T17:10:34","date_gmt":"2024-10-10T17:10:34","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5298"},"modified":"2026-04-02T17:33:04","modified_gmt":"2026-04-02T17:33:04","slug":"analysing-social-media-sentiment-with-ibm-event-processing","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5298","title":{"rendered":"Analysing social media sentiment with IBM Event Processing"},"content":{"rendered":"<h3>aka \u201cWho wants a Mario alarm clock?\u201d<\/h3>\n<p><strong>In this post, I want to share a quick demo of using Event Processing to process social media posts.<\/strong><\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/overview.png\" alt=\"diagram\"\/><\/p>\n<h3>Background<\/h3>\n<p>A fun surprise from Nintendo today: they&#8217;ve introduced a new product! <a href=\"https:\/\/www.nintendo.com\/en-gb\/Hardware\/Nintendo-Sound-Clock-Alarmo\/Nintendo-Sound-Clock-Alarmo-2670177.html\">\u201cAlarmo\u201d<\/a> is a game-themed alarm clock, with some interesting gesture recognition features.<\/p>\n<p>I was (unsurprisingly!) tempted&#8230;<\/p>\n<p><iframe src=\"https:\/\/mastodon.org.uk\/@dalelane\/113282624377758691\/embed\" class=\"mastodon-embed\" style=\"max-width: 100%; border: 0\" width=\"400\" allowfullscreen=\"allowfullscreen\"><\/iframe><script src=\"https:\/\/mastodon.org.uk\/embed.js\" async=\"async\"><\/script><\/p>\n<p>But that got me wondering how the rest of the Internet was reacting.<\/p>\n<p>In this post, I want to share a (super-simple!) demo for how to look at this &#8211; using IBM Event Processing to create an Apache Flink job that looks at the sentiment of social media posts about this unusual new product.<\/p>\n<p><!--more-->This is what I set up. I&#8217;ll share a few details for each of the 4 steps in case this inspires you to try something similar.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/overview.png\" alt=\"diagram\"\/><\/p>\n<h3>Step 1 &#8211; Getting social media posts into Kafka<\/h3>\n<p>I&#8217;m using <a href=\"https:\/\/joinmastodon.org\">Mastodon<\/a> for this, as it has <a href=\"https:\/\/docs.joinmastodon.org\/api\/\">a friendly and open API<\/a>. I configured <a href=\"https:\/\/github.com\/dalelane\/kafka-connect-mastodon-source\">a Kafka Connect connector<\/a> to bring mastodon posts with the <code style=\"color: #770000; font-weight: bold;\">#nintendo<\/code> hashtag into Kafka.<\/p>\n<p>I added a Mastodon connector jar to <a href=\"https:\/\/github.com\/IBM\/event-automation-demo\/blob\/main\/install\/eventstreams\/templates\/07-kafkaconnect.yaml\">the Kafka Connect runtime I use for demos<\/a>.<\/p>\n<pre style=\"border: thin #AA0000 solid; color: #770000; background-color: #ffffc0; padding: 1em; overflow-y: scroll; overflow-x: scroll; font-size: 0.8em; white-space: pre; max-height: 400px;\">apiVersion: eventstreams.ibm.com\/v1beta2\nkind: KafkaConnect\nmetadata:\n  annotations:\n    eventstreams.ibm.com\/use-connector-resources: 'true'\n  name: kafka-connect-cluster\nspec:\n  ...\n  build:\n    ...\n    plugins:\n      - artifacts:\n          - type: jar\n            url: 'https:\/\/github.com\/dalelane\/kafka-connect-mastodon-source\/releases\/download\/0.0.1\/kafka-connect-mastodon-source-0.0.1-jar-with-dependencies.jar'\n        name: mastodon\n      ...\n<\/pre>\n<p>I created a KafkaConnector instance to start a new connector:<\/p>\n<pre style=\"border: thin #AA0000 solid; color: #770000; background-color: #ffffc0; padding: 1em; overflow-y: scroll; overflow-x: scroll; font-size: 0.8em; white-space: pre; max-height: 400px;\">apiVersion: eventstreams.ibm.com\/v1beta2\nkind: KafkaConnector\nmetadata:\n  name: mastodon-nintendo\n  namespace: event-automation\n  labels:\n    eventstreams.ibm.com\/cluster: kafka-connect-cluster\nspec:\n  class: uk.co.dalelane.kafkaconnect.mastodon.source.MastodonSourceConnector\n  config:\n    key.converter: org.apache.kafka.connect.storage.StringConverter\n    key.converter.schemas.enable: false\n    value.converter: org.apache.kafka.connect.json.JsonConverter\n    value.converter.schemas.enable: false\n    mastodon.accesstoken: MySuperSecretAccessTokenForMastodonAPI\n    mastodon.instance: mastodon.social\n    mastodon.searchterm: nintendo\n    mastodon.topic: mastodon\n<\/pre>\n<p>That gave me with a Kafka topic with Mastodon social media posts about Nintendo.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/kafka-topic.png\"\/><\/p>\n<h3>Step 2 &#8211; Processing the posts<\/h3>\n<p>I used <a style=\"font-weight: bold;\" href=\"https:\/\/www.ibm.com\/products\/event-automation\/event-processing\">IBM Event Processing<\/a> to create an Apache Flink job to start processing these events.<\/p>\n<p>I added filters to limit the posts to the ones I&#8217;m interested in.<\/p>\n<p>For example, I filtered out posts by bots, as I was curious to see what people were organically saying about the product, not just auto-posted links.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/filter-bots.png\"\/><\/p>\n<p>And I added a filter to only keep posts that mentioned something to do with this new product.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/filter-mentions.png\"\/><\/p>\n<p>That gave me a starting point with a stream of posts worth looking at.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/filtered-posts.png\"\/><\/p>\n<h3>Step 3 &#8211; Measure the sentiment of the comments<\/h3>\n<p>I used <a href=\"https:\/\/www.ibm.com\/products\/natural-language-understanding\" style=\"font-weight: bold;\">IBM Watson Natural Language Understanding<\/a> for this.<\/p>\n<p>I created an instance of the service using the Lite plan as that let me get started for free.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/nlu-instance.png\"\/><\/p>\n<p>I downloaded the OpenAPI definition for the API from <a href=\"https:\/\/cloud.ibm.com\/apidocs\/natural-language-understanding\">the Watson NLU documentation page<\/a> (<em>using the three-dot menu in the top-left<\/em>).<\/p>\n<p><em>Aside: There&#8217;s a tiny bug I needed to fix in the spec &#8211; replacing the incorrect &#8220;apikey&#8221; security scheme in the doc to match what the Watson API actually expects:<\/em><\/p>\n<pre style=\"border: thin #AA0000 solid; color: #770000; background-color: #ffffc0; padding: 1em; overflow-y: scroll; overflow-x: scroll; font-size: 0.8em; white-space: pre; max-height: 400px;\">    ...\n    \"securitySchemes\": {\n      \"IAM\": {\n        \"type\": \"http\",\n        \"scheme\": \"basic\"\n      }\n    },\n    ...\n<\/pre>\n<p>I could then use this API in Event Processing to enrich each event with the Watson NLU service&#8217;s assessment of the sentiment:<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/api-definition.png\"\/><\/p>\n<p>(The API key can be found in the &#8220;Service credentials&#8221; tab of the Watson Natural Language Understanding service instance page.)<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/api-security.png\"\/><\/p>\n<p>The input mapping page is pretty long as the Watson service has a <strong>lot<\/strong> of options, but following <a href=\"https:\/\/cloud.ibm.com\/apidocs\/natural-language-understanding\">the API doc<\/a> makes that easy.<\/p>\n<p>I set the version of the service to use as shown in the examples in the API docs, and I enabled the sentiment analysis feature. (The service can return lots of other details about the text, such as emotion, entity extraction, relationships, etc. &#8211; which is why there are so many options here.)<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/api-input.png\"\/><\/p>\n<p>The Mastodon API returns the HTML content for social media posts, so I put that into the <code style=\"color: #770000; font-weight: bold;\">html<\/code> property.<\/p>\n<p>Posts are written by people in a wide variety of languages, but the Watson NLU service can handle that. I included the language code from the Mastodon post in the request to the Watson service so it could analyse the text appropriately.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/api-input-2.png\"\/><\/p>\n<p>There is a lot of output you can get from the service, but I was most interested in the sentiment analysis score and label.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/api-output.png\"\/><\/p>\n<h3>Step 4 &#8211; Outputting the results<\/h3>\n<p>At this point, there are lots of things you can do with the output from this.<\/p>\n<p>You can just have the raw results, with the posts together with the analysis of their sentiment.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/flow-results-raw.png\"\/><\/p>\n<p>Or you can aggregate the posts &#8211; maybe try computing something with those sentiment scores, or even just keep it simple and count the number of posts with each sentiment label.<\/p>\n<p><img decoding=\"async\" style=\"width: 100%; max-width: 600px; border: thin black solid;\" src=\"https:\/\/images.dalelane.co.uk\/2024-10-10-flinksocial\/flow-results-agg.png\"\/><\/p>\n<h3>Summary<\/h3>\n<p>I&#8217;m not sure I&#8217;m any closer to deciding if I need a Nintendo-themed gesture-controlled alarm clock.<\/p>\n<p>But hopefully I&#8217;ve at least demonstrated how easy it is to take a stream of events from social media, and enrich them using IBM Watson services to gain insight into public discussions.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>aka \u201cWho wants a Mario alarm clock?\u201d In this post, I want to share a quick demo of using Event Processing to process social media posts. Background A fun surprise from Nintendo today: they&#8217;ve introduced a new product! \u201cAlarmo\u201d is a game-themed alarm clock, with some interesting gesture recognition features. I was (unsurprisingly!) tempted&#8230; But [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":5302,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,583,584],"class_list":["post-5298","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-apachekafka","tag-ibmeventstreams","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5298","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5298"}],"version-history":[{"count":2,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5298\/revisions"}],"predecessor-version":[{"id":5959,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5298\/revisions\/5959"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/5302"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5298"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5298"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5298"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}