I’ve submitted a Flink Improvement Proposal (FLIP) about adding support for Flink to be able to fetch remote jars for applications submitted to a session cluster.
Archive for August, 2026
Adding support for remote jar fetching in Flink session clusters
Sunday, August 23rd, 2026Why we can’t have nice things
Sunday, August 9th, 2026Looking at the server request logs for Machine Learning for Kids is a depressing reminder of what it means to run something on the Internet nowadays.
I was skimming through the request logs for the Machine Learning for Kids servers yesterday [1], and what jumped out at me was the constant rate of requests to non-existent paths that I just generate HTTP 404 responses for.
These aren’t someone accidentally mistyping a URL.
I grepped an excerpt from the log for requests that got an HTTP-404 response and put the result in a gist.
Importing and exporting projects in Machine Learning for Kids
Saturday, August 8th, 2026I’ve added new features to Machine Learning for Kids to give students more options about where they store their projects.
Limits
In Machine Learning for Kids, students create projects where they collect training examples to train their own custom machine learning models. I don’t limit the number of projects students can create. I want students to create as many projects as they’d like. But for projects stored on my server, I’ve nudged students to delete projects once they are no longer working on them. This is intentional, as a way of helping me keep the running costs for the site under control.
Running Flink jobs in MiniCluster using the Kubernetes Operator
Wednesday, August 5th, 2026I’ve submitted a Flink Improvement Proposal (FLIP) about adding support for the Flink Kubernetes Operator to run Flink’s MiniCluster in a single pod for low-throughput jobs that require isolation.
In the last few weeks, I’ve been working on a proof-of-concept to demonstrate the feasibility of this idea. I’ve done enough to convince myself that this is viable and identify where the issues will be, but I’m looking for community feedback before I take it much further.
Background
The Flink Kubernetes Operator is one of the best ways to run Flink jobs. From the documentation:
Flink deployments are declared like any other Kubernetes workload, and the operator runs their whole operational life:
- Lifecycle Management: deployment, stateful upgrades, rollbacks, and self-healing
- Zero-Downtime Upgrades: blue/green deployments that switch over only once the new version is proven healthy
- Autoscaling: parallelism and memory continuously right-sized to the observed load
- Kubernetes-Native Operations: Helm installation, RBAC, high availability, metrics, logging, and ingress
You create a FlinkDeployment Kubernetes custom resource that points to your Flink application, and the Operator handles provisioning independently schedulable and independently configurable Job Manager Deployment and Task Manager Deployment, configuring them to form a distributed Flink cluster.
This provides scalability and high availability, and is the right approach in a lot of situations. However, this comes at the cost of a fixed baseline cost of at least one JobManager pod and one or more TaskManager pods.
A smaller, lighter-weight alternative would be useful for small or intermittent jobs, where the minimum resource cost of two separately-scheduled pods is disproportionate to the job itself.
A single-pod, self-contained Flink job that starts fast and needs no multi-pod coordination could be a good fit for low-throughput jobs that aren’t suitable for session clusters because they need isolation.