{"id":5477,"date":"2025-01-28T22:36:14","date_gmt":"2025-01-28T22:36:14","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=5477"},"modified":"2026-03-14T21:29:02","modified_gmt":"2026-03-14T21:29:02","slug":"turning-noise-into-actionable-alerts-using-flink","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=5477","title":{"rendered":"Turning noise into actionable alerts using Flink"},"content":{"rendered":"

In this post, I want to share two examples of how you can use pattern recognition in Apache Flink to turn a noisy stream of output into something useful and actionable.<\/strong><\/p>\n

Imagine that you have a Kafka topic with a stream of events that you sometimes need to respond to.<\/p>\n

You can think of this conceptually as being a stream of values that could be plotted against time.<\/p>\n

<\/a><\/p>\n

To make this less abstract, we can use the events from the sensor readings topic that the \u201cLoosehanger\u201d data generator<\/a> produces to. Those events are a stream of temperature and humidity readings.<\/p>\n

<\/a><\/p>\n

Imagine that these events represent something that you might need to respond to when the event for a sensor exceeds some given threshold.<\/p>\n

You can think of it visually like this:<\/p>\n

<\/a><\/p>\n

When the events contain values that are over the threshold, this is something that you care about. This is something you need to respond to.<\/p>\n

You might just think of that as a filtering use case.<\/p>\n

For the \u201cLoosehanger\u201d sensor readings, that could be filtering on sensor readings with temperature or humidity readings that exceed a threshold.<\/p>\n

In Flink SQL, that would be something like this:<\/p>\n

SELECT\n    *\nFROM\n    `ALL SENSORS`\nWHERE\n    temperature > 23.5\n        OR\n    humidity > 58;<\/pre>\n
If you use the Event Processing UI<\/a> to create that, it would look like this:<\/p>\n
<\/a><\/p>\n
A Flink job like this will output all events with values that indicate something needs attention.<\/p>\n
<\/a><\/p>\n
The result of a processing flow like this is likely a large and \u201cnoisy\u201d burst of output throughout the time while something needs attention.<\/p>\n
<\/a><\/p>\n
Is that helpful output?<\/p>\n
After the first event, are the next hundred events you get still useful?<\/p>\n
<\/a><\/p>\n
Probably not.<\/p>\n
You probably want output that looks more like this:<\/p>\n
<\/a><\/p>\n
You want to know when the events indicate that something has happened that needs your attention. While you\u2019re responding to that, you don\u2019t want to be disturbed with more notifications until the next time it happens.<\/p>\n
There are several different ways you can implement this. Here are two easy ways to approach it.<\/p>\n
approach 1 : time-based<\/h3>\n
After emitting an actionable alert, you could ignore any subsequent events (that contain values over the threshold) for a fixed time interval.<\/p>\n
<\/a><\/p>\n
After an actionable alert, you can assume that the issue will be given attention. You can assume someone will look at it and fix it. This means there is no need to trigger additional alerts for a short time afterwards.<\/p>\n
For example, doing this with the \u201cLoosehanger\u201d sensor events using a time interval of 10 minutes:<\/p>\n
CREATE TEMPORARY VIEW MATCHREC AS\n  SELECT\n    *\n  FROM\n    `HIGH VALUE READINGS`\n\n    MATCH_RECOGNIZE (\n      PARTITION BY<\/strong>\n        sensorid\n      ORDER BY<\/strong>\n        event_time\n      MEASURES<\/strong>\n        LAST(AFTER_TIME_INTERVAL.event_time)  AS event_time,\n        LAST(AFTER_TIME_INTERVAL.temperature) AS temperature,\n        LAST(AFTER_TIME_INTERVAL.humidity)    AS humidity\n      ONE ROW PER MATCH\n      AFTER MATCH SKIP TO LAST AFTER_TIME_INTERVAL\n      PATTERN<\/strong> (\n        LAST_ALERT   DURING_TIME_INTERVAL*   AFTER_TIME_INTERVAL\n      )\n      DEFINE<\/strong>\n        DURING_TIME_INTERVAL AS\n          event_time <= TIMESTAMPADD(MINUTE, 10, LAST_ALERT.event_time),\n        AFTER_TIME_INTERVAL  AS\n          event_time >  TIMESTAMPADD(MINUTE, 10, LAST_ALERT.event_time)\n);<\/pre>\n
<\/a><\/p>\n
This is a pattern that you want Flink to recognize:<\/p>\n