I’ve made a Kafka Connect source connector for sending “real-time” (not really) stock price events to a Kafka topic.
Choose your stock (e.g. “IBM“) and run the Connector pointed at your topic, and every minute a JSON event will be produced like:
{"open":137.9,"high":137.9,"low":137.9,"close":137.9,"volume":500,"timestamp":1629421200,"datetime":"2021-08-19 20:00:00"}
It’s a quick and simple hack I needed for a proof-of-concept based on processing live stock price events. But I wanted something I could run with a free API key so it couldn’t use anything really “live” live.
I found Alpha Vantage who have a few simple APIs, including a time-series API that returns historical stock price data in one-minute intervals. They offer free API keys that allow up to 500 requests a day.
My Connector is using this to… cheat a little.
The Connector uses Alpha Vantage’s time-series API to download a history of one-minute interval time-series records. It caches them and then gradually produces them to the Kafka topic – each event time-shifted by a configurable number of hours.
For example, if you set the delay to 24, it downloads the last day of stock price records, then sends each stock price event to Kafka twenty-four-hours after the timestamp in the price record.
This means the Connector only needs to make one API call a day, which is easily within the free API tier limits. But it’s still able to produce an event every minute, so it looks enough like real-time stock price events for my proof-of-concept project.
By default, I set the delay to 168 hours (one week) to make it look even more realistic (for example, it means events are correctly only published on weekdays and not weekends).
Just a quick hack, but I thought I’d share it. My use case is a little unusual, but if you also need (something that looks a little like a realistic) real-time stock-price events in Kafka, then you can find it on github.
Tags: apachekafka, kafka