Posts Tagged ‘kafka’

An introduction to serverless and OpenWhisk for Kafka users

Saturday, July 13th, 2019

I gave a talk at Kafka Summit London this year about Apache OpenWhisk. It was aimed at Kafka users who want to know what the serverless hype is all about.

I covered:

  • a simple introduction of what serverless is for
  • an introduction to some of the serverless platforms available
  • a quick crash course in how to get started with Apache OpenWhisk

I also had a quick tangent looking into how Apache OpenWhisk itself uses Kafka internally, because I thought that was interesting!

My slides are on SlideShare if you’d like to see a higher-res version of any of them.

If this convinces you to give OpenWhisk a try, I have a post on how to get started with OpenWhisk that has all the commands you need to copy/paste to get yourself a working OpenWhisk environment connected to a Kafka source of events.

(more…)

Getting started with OpenWhisk and Kafka

Saturday, July 6th, 2019

Apache OpenWhisk (and serverless platforms in general) are a great way to host and manage code that you want to run in response to events.
Apache Kafka topics are a great source of events.

In this post, I’ll run through a super simple beginner’s guide to writing code for OpenWhisk that processes events on your Kafka topics.

(more…)

Using Node-RED with IBM Event Streams

Friday, June 28th, 2019


Click to enlarge

IBM Event Streams is the distributed streaming real-time data platform Apache Kafka, from IBM.

Node-RED is a visual flow-based development tool, with nodes that you drag and drop onto a canvas and wire together. It’s useful for loads of tasks, such as quick and flexible prototyping.

In this post, I’ll show how Event Streams and Node-RED work well together. You can use Node-RED to quickly and easily create flows that consume messages from Kafka topics, or that process events from different sources and produce the output to Kafka topics.

(more…)

Using kafkacat and kaf with IBM Event Streams

Sunday, June 9th, 2019

IBM Event Streams is IBM’s Kafka offering. Naturally it comes with it’s own UI and CLI tools, but one of the great things about Apache Kafka is that it’s not just a single thing from a single company – rather it is an active and diverse ecosystem, which means you’ve got a variety of tools to choose from.

I thought I’d try a couple of open source CLI tools, and share how to connect them and what they can do.

First up, kafkacat.

(more…)

Setting up Slack alerts to monitor IBM Event Streams

Sunday, October 7th, 2018

IBM Event Streams brings Apache Kafka to IBM Cloud Private (together with a bunch of other useful stuff to make it easier to run and use Kafka).

Monitoring is an important part of running a Kafka cluster. There are a variety of metrics that are useful indicators of the health of the cluster and serve as warnings of potential future problems.

To that end, Event Streams collects metrics from all of the Kafka brokers and exports them to a Prometheus-based monitoring platform.

There are three ways to use this:

1) A selection of metrics can be viewed from a dashboard in the Event Streams admin UI.
eventstreams-monitoring-20181006-11
This is good for a quick way to get started.

2) Grafana is pre-configured and available out-of-the-box to create custom dashboards
eventstreams-monitoring-20181006-13
This will be useful for long-term projects, as Grafana lets you create dashboards showing the metrics that are most important for your unique needs. A sample dashboard is included to help get you started.

3) Alerts can be created, so that metrics that meet predefined criteria can be used to push notifications to a variety of tools, like Slack, PagerDuty, HipChat, OpsGenie, email, and many, many more.
eventstreams-monitoring-20181006-20
This is useful for being able to respond to changes in the metrics values when you’re not looking at the Monitor UI or Grafana dashboard.

For example, you might want a combination of alert approaches like:

  • metrics and/or metric values that might not be urgent but should get some attention result in an automated email being sent to a team email address
  • metrics and/or metric values that suggest a more severe issue could result in a Slack message to a team workspace
  • metrics and/or metric values that suggest an urgent critical issue could result in creating a PagerDuty ticket so that it gets immediate attention

This post is about this third use of monitoring and metrics: how you can configure alerts based on the metrics available from your Kafka brokers in IBM Event Streams.

(more…)