Building the Largest Kubernetes Cluster: 130k Nodes & Beyond

The official upstream documentation states that a single Kubernetes Cluster supports up to 5,000 nodes. For the average enterprise, this is overkill. For hyperscalers and platform engineers designing the next generation of cloud infrastructure, it’s merely a starting point.

When we talk about managing a fleet of 130,000 nodes, we enter a realm where standard defaults fail catastrophically. We are no longer just configuring software; we are battling the laws of physics regarding network latency, etcd storage quotas, and Go routine scheduling. This article dissects the architectural patterns, kernel tuning, and control plane sharding required to push a Kubernetes Cluster (or a unified fleet of clusters) to these extreme limits.

The “Singularity” vs. The Fleet: Defining the 130k Boundary

Before diving into the sysctl flags, let’s clarify the architecture. Running 130k nodes in a single control plane is currently theoretically impossible with vanilla upstream Kubernetes due to the etcd hard storage limit (8GB recommended max) and the sheer volume of watch events.

Achieving this scale requires one of two approaches:

1. The “Super-Cluster” (Heavily Modified): Utilizing sharded API servers and segmented etcd clusters (splitting events from resources) to push a single cluster ID towards 10k–15k nodes.
2. The Federated Fleet: Managing 130k nodes across multiple clusters via a unified control plane (like Karmada or custom controllers) that abstracts the “cluster” concept away from the user.

We will focus on optimizing the unit—the Kubernetes Cluster—to its absolute maximum, as these optimizations are prerequisites for any large-scale fleet.

Phase 1: Surgical Etcd Tuning

At scale, etcd is almost always the first bottleneck. In a default Kubernetes Cluster, etcd stores both cluster state (Pods, Services) and high-frequency events. At 10,000+ nodes, the write IOPS from Kubelet heartbeats and event recording will bring the cluster to its knees.

1. Vertical Sharding of Etcd

You must split your etcd topology. Never run events in the same etcd instance as your cluster configuration.

# Example API Server flags to split storage
--etcd-servers="https://etcd-main-0:2379,https://etcd-main-1:2379,..."
--etcd-servers-overrides="/events#https://etcd-events-0:2379,https://etcd-events-1:2379,..."

2. Compression and Quotas

The default 2GB quota is insufficient. Increase the backend quota to 8GB (the practical safety limit). Furthermore, enable compression in the API server to reduce the payload size hitting etcd.

Pro-Tip: Monitor the etcd_mvcc_db_total_size_in_bytes metric religiously. If this hits the quota, your cluster enters a read-only state. Implement aggressive defragmentation schedules (e.g., every hour) for the events cluster, as high churn creates massive fragmentation.

Phase 2: The API Server & Control Plane

The kube-apiserver is the CPU-hungry brain. In a massive Kubernetes Cluster, the cost of serialization and deserialization (encoding/decoding JSON/Protobuf) dominates CPU cycles.

Priority and Fairness (APF)

Introduced to prevent controller loops from dDoSing the API server, APF is critical at scale. You must define custom FlowSchemas and PriorityLevelConfigurations. The default “catch-all” buckets will fill up instantly with 10k nodes, causing legitimate administrative calls (`kubectl get pods`) to time out.

apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind: PriorityLevelConfiguration
metadata:
  name: system-critical-high
spec:
  type: Limited
  limited:
    assuredConcurrencyShares: 50
    limitResponse:
      type: Queue

Disable Unnecessary API Watches

Every node runs a kube-proxy and a kubelet. If you have 130k nodes, that is 130k watchers. If a significantly scoped change happens (like an EndpointSlice update), the API server must serialize that update 130k times.

• Optimization: Use EndpointSlices instead of Endpoints.
• Optimization: Set --watch-cache-sizes manually for high-churn resources to prevent cache misses which force expensive calls to etcd.

Phase 3: The Scheduler Throughput Challenge

The default Kubernetes scheduler evaluates every feasible node to find the “best” fit. With 130k nodes (or even 5k), scanning every node is O(N) complexity that results in massive scheduling latency.

You must tune the percentageOfNodesToScore parameter.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
percentageOfNodesToScore: 5  # Only look at 5% of nodes before making a decision

By lowering this to 5% (or even less in hyperscale environments), you trade a theoretical “perfect” placement for the ability to actually schedule pods in a reasonable timeframe.

Phase 4: Networking (CNI) at Scale

In a massive Kubernetes Cluster, iptables is the enemy. It relies on linear list traversal for rule updates. At 5,000 services, iptables becomes a noticeable CPU drag. At larger scales, it renders the network unusable.

IPVS vs. eBPF

While IPVS (IP Virtual Server) uses hash tables and offers O(1) complexity, modern high-scale clusters are moving entirely to eBPF (Extended Berkeley Packet Filter) solutions like Cilium.

• Why: eBPF bypasses the host networking stack for pod-to-pod communication, significantly reducing latency and CPU overhead.
• Identity Management: At 130k nodes, storing IP-to-Pod mappings is expensive. eBPF-based CNIs can use identity-based security policies rather than IP-based, which scales better in high-churn environments.

Phase 5: The Node (Kubelet) Perspective

Often overlooked, the Kubelet itself can dDoS the control plane.

• Heartbeats: Adjust --node-status-update-frequency. In a 130k node environment (likely federated), you do not need 10-second heartbeats. Increasing this to 1 minute drastically reduces API server load.
• Image Pulls: Serialize image pulls (`–serialize-image-pulls=false`) carefully. While parallel pulling is faster, it can spike disk I/O and network bandwidth, causing the node to go NotReady under load.

Frequently Asked Questions (FAQ)

What is the hard limit for a single Kubernetes Cluster?

As of Kubernetes v1.29+, the official scalability thresholds are 5,000 nodes, 150,000 total pods, and 300,000 total containers. Exceeding this requires significant customization of the control plane, specifically around etcd storage and API server caching mechanisms.

How do Alibaba and Google run larger clusters?

Tech giants often run customized versions of Kubernetes. They utilize techniques like “Cell” architectures (sharding the cluster into smaller failure domains), custom etcd storage drivers, and highly optimized networking stacks that replace standard Kube-Proxy implementations.

Should I use Federation or one giant cluster?

For 99% of use cases, Federation (multi-cluster) is superior. It provides better isolation, simpler upgrades, and drastically reduces the blast radius of a failure. Managing a single Kubernetes Cluster of 10k+ nodes is a high-risk operational endeavor.

Conclusion

Building a Kubernetes Cluster that scales toward the 130k node horizon is less about installing software and more about systems engineering. It requires a deep understanding of the interaction between the etcd key-value store, the Go runtime scheduler, and the Linux kernel networking stack.

While the allure of a single massive cluster is strong, the industry best practice for reaching this scale involves a sophisticated fleet management strategy. However, applying the optimizations discussed here-etcd sharding, APF tuning, and eBPF networking-will make your clusters, regardless of size, more resilient and performant. Thank you for reading the DevopsRoles page!