Posts Tagged ‘apachekafka’

« Older Entries
Newer Entries »

IBM Event Streams v10

Tuesday, June 30th, 2020

On Friday, we released the latest version of IBM Event Streams. This means I’ve been doing a variety of demo sessions to show people what we’ve made and how it works.

Here’s a recording of one of them:

In this session, I did a run-through of the new Event Streams Operator on Red Hat OpenShift, with a very quick intro to some of the features:

00m30s – installing the Operator
02m10s – creating custom Kafka clusters in the OpenShift console
05m10s – creating custom Kafka clusters in IBM Cloud Pak for Integration
08m00s – running the sample Kafka application
08m50s – creating topics
10m20s – creating credentials for client applications
11m45s – automating deployment of event-streaming infrastructure
12m30s – using schemas with the schema registry
13m10s – sending messages with HTTP POST requests
13m45s – viewing messages in the message browser
14m00s – command line administration
14m30s – running Kafka Connect
15m10s – geo-replication for disaster recovery
15m50s – monitoring Kafka clusters in the Event Streams UI
17m10s – monitoring with custom Grafana dashboards
17m30s – alerting using Prometheus

Tags:, , , ,
Posted in ibm | Comments Closed

Why are Kafka messages still on the topic after the retention time has expired?

Sunday, February 9th, 2020

We had an interesting Kafka question from an Event Streams user. The answer isn’t immediately obvious unless you know a bit about Kafka internals, and after a little searching I couldn’t find an explanation online, so I thought I’d share the answer here (obviously anonymised and heavily simplified).

What is retention?

Retention is a Kafka feature to help you manage the amount of disk space your topics use.

It lets you specify how long you want Kafka to keep messages on a topic for. You can specify this by time (e.g. “I want messages on this topic to be preserved for at least X days”) or by disk usage (e.g. “I want at least the last X gb of messages on this topic to be preserved”).

After the retention time or disk threshold is exceeded, messages become eligible for being automatically deleted by Kafka.

What was wrong in this case?

screenshot

They had created a topic with a retention time of 7 days.

They had assumed that this meant messages older than 7 days would be deleted.

When they looked at the messages on their topic, they could see some messages older than 7 days were there, and were surprised.

They thought this might mean retention wasn’t working.

(more…)

Tags:, ,
Posted in tech | 4 Comments »

Using TensorFlow with IBM Event Streams
(Kafka + Machine Learning = Awesome)

Thursday, October 31st, 2019

In this post, I want to explain how to get started creating machine learning applications using the data you have on Kafka topics.

I’ve written a sample app, with examples of how you can use Kafka topics as:

  • a source of training data for creating machine learning models
  • a source of test data for evaluating machine learning models
  • an ongoing stream of events to make predictions about using machine learning models

I’ll use this post to explain how it works, and how you can use it as the basis of writing your first ML pipeline using the data on your own Kafka topics.

(more…)

Tags:, , , ,
Posted in code | Comments Closed

Using Avro schemas from Python apps with IBM Event Streams

Thursday, October 17th, 2019

I’ve written before about how to write a schema for your developers using Kafka. The examples I used before were all in Java, but someone asked me yesterday if I could share some Python equivalents.

The principles are described in the Event Streams documentation, but in short, your Kafka producers use Apache Avro to serialize the message data that you send, and identify the schema that you’ve used in the Kafka message header. In your Kafka consumers, you look at the headers of the messages that you receive to know which schema to retrieve, and use that to deserialize message data.

(more…)

Tags:, , , , ,
Posted in code | Comments Closed

SQL queries on Kafka topics using Apache Hive

Tuesday, August 6th, 2019

Apache Hive is open source data warehouse software built on top of Hadoop. It gives you an SQL-like interface to a wide variety of databases, filesystems, and other systems.

One of the Hive storage handlers is a Kafka storage handler, which lets you create a Hive “external table” based on a Kafka topic.

And once you’ve created a Hive table based on a Kafka topic, you can run SQL queries based on attributes of the messages on that topic.

I was having a play with Hive this evening, as a way of running SQL queries against messages on my Kafka topics. In this post, I’ll share a few queries that I tried.


(more…)

Tags:, , , , ,
Posted in code | Comments Closed

How to write your first Avro schema

Saturday, July 20th, 2019

Any time there is more than one developer using a Kafka topic, they will need a way to agree on the shape of the data that will go into messages. The most common way to document the schema of messages in Kafka is to use the Apache Avro serialization system.

This post is a beginner’s guide to writing your first Avro schema, and a few tips for how to use it in your Kafka apps.

(more…)

Tags:, , , ,
Posted in code | 2 Comments »

An introduction to serverless and OpenWhisk for Kafka users

Saturday, July 13th, 2019

I gave a talk at Kafka Summit London this year about Apache OpenWhisk. It was aimed at Kafka users who want to know what the serverless hype is all about.

I covered:

  • a simple introduction of what serverless is for
  • an introduction to some of the serverless platforms available
  • a quick crash course in how to get started with Apache OpenWhisk

I also had a quick tangent looking into how Apache OpenWhisk itself uses Kafka internally, because I thought that was interesting!

My slides are on SlideShare if you’d like to see a higher-res version of any of them.

If this convinces you to give OpenWhisk a try, I have a post on how to get started with OpenWhisk that has all the commands you need to copy/paste to get yourself a working OpenWhisk environment connected to a Kafka source of events.

(more…)

Tags:, , ,
Posted in code | Comments Closed

Getting started with OpenWhisk and Kafka

Saturday, July 6th, 2019

Apache OpenWhisk (and serverless platforms in general) are a great way to host and manage code that you want to run in response to events.
Apache Kafka topics are a great source of events.

In this post, I’ll run through a super simple beginner’s guide to writing code for OpenWhisk that processes events on your Kafka topics.

(more…)

Tags:, ,
Posted in code | Comments Closed

« Older Entries
Newer Entries »