I’ve submitted a Flink Improvement Proposal (FLIP) about adding support for the Flink Kubernetes Operator to run Flink’s MiniCluster in a single pod for low-throughput jobs that require isolation.
In the last few weeks, I’ve been working on a proof-of-concept to demonstrate the feasibility of this idea. I’ve done enough to convince myself that this is viable and identify where the issues will be, but I’m looking for community feedback before I take it much further.
Background
The Flink Kubernetes Operator is one of the best ways to run Flink jobs. From the documentation:
Flink deployments are declared like any other Kubernetes workload, and the operator runs their whole operational life:
- Lifecycle Management: deployment, stateful upgrades, rollbacks, and self-healing
- Zero-Downtime Upgrades: blue/green deployments that switch over only once the new version is proven healthy
- Autoscaling: parallelism and memory continuously right-sized to the observed load
- Kubernetes-Native Operations: Helm installation, RBAC, high availability, metrics, logging, and ingress
You create a FlinkDeployment Kubernetes custom resource that points to your Flink application, and the Operator handles provisioning independently schedulable and independently configurable Job Manager Deployment and Task Manager Deployment, configuring them to form a distributed Flink cluster.
This provides scalability and high availability, and is the right approach in a lot of situations. However, this comes at the cost of a fixed baseline cost of at least one JobManager pod and one or more TaskManager pods.
A smaller, lighter-weight alternative would be useful for small or intermittent jobs, where the minimum resource cost of two separately-scheduled pods is disproportionate to the job itself.
A single-pod, self-contained Flink job that starts fast and needs no multi-pod coordination could be a good fit for low-throughput jobs that aren’t suitable for session clusters because they need isolation.
MiniClusters
Apache Flink has a mechanism suited to this: MiniCluster.
MiniCluster is an in-JVM cluster that runs a real JobManager and one or more TaskManagers merged into a single Java process. It instantiates the same component classes as a full distributed deployment, meaning that job graphs, checkpoints, save points, state backends, connectors, metrics, and more are all fully compatible with a normal distributed Flink cluster.
Users who could use this to run very small Flink deployments in Kubernetes currently need to deploy this themselves as a bare MiniCluster in a hand-rolled pod, losing the benefits of the operator’s lifecycle management, status reporting, and savepoint / checkpoint tooling.
The FLIP
I’ve been working on how the Operator could provide first-class support for running a job in MiniCluster, which I’ve described here.
You’d create my new FlinkMiniCluster type of Kubernetes custom resource, pointing to the exact same Flink application, and the Operator will instead deploy your application in a single pod topology. You still get the same declarative lifecycle management, status reporting, and snapshot tooling you have for Flink jobs in Kubernetes today, but with a much smaller footprint.
The trade-off for running everything in a single process in a single pod is that any fatal error will take down your whole merged Flink MiniCluster. There is no live failover. But other Flink features such as savepoints and checkpoints mean that FlinkMiniCluster jobs will still have support for crash-consistent resume, and the simplification of running in a single process means restart after a crash is very fast. For some Flink jobs, I think this trade-off will be acceptable.
What next?
Time to see if the Flink community feel like this would be useful.
I’ve written up a more detailed proposal in a Google Doc and asked for feedback on the Flink dev mailing list.
If you have opinions on any of this, please do join the discussion – or you can let me know directly.