At the recent DevOpsDays Tashkent conference, Artem Grinberg, Head of Cloud Products at UzCloud, presented a report on the internal architecture of modern container platforms. The presentation was based on his personal experience scaling infrastructure up to 4,000 active clusters. We are currently integrating these proven engineering approaches into the UzCloud architecture to ensure maximum stability for client environments.
In this article, we examine the key points of the presentation, explain the logic behind Managed Kubernetes services, discuss the specifics of designing fault-tolerant systems, and look at the tool that helps manage a large fleet of isolated environments effectively.
What Managed Kubernetes Is and What Problems It Solves
The Kubernetes platform has become the industry standard for automating the deployment, scaling, and management of containerized applications. However, maintaining such an infrastructure independently requires deep expertise and a constant time commitment from an IT team. Engineers must manually configure the control plane, network setups, monitoring systems, updates, and baseline security.
A managed service (Managed Kubernetes) shifts routine administration tasks to the infrastructure provider. The provider handles deployment, control plane fault tolerance, regular updates, and 24/7 monitoring. As a result, your technical team receives a ready-to-use environment and can focus on writing code and developing the product without distractions from server support.
The Scaling Challenge: Why the Classic Approach Loses Efficiency
With a small number of managed clusters, engineers typically use a centralized push model. In this setup, commands to create, modify, or delete resources are sent from a single internal administration control plane directly to client environments.
This approach works reliably until the platform faces a surge in automatic scaling (autoscaling) tasks on the user side. When loads change within a client application, the autoscaling system triggers a long chain of background processes:
- Allocating new virtual machines.
- Recalculating network routes.
- Updating configurations.
- Updating metadata.
When these processes occur simultaneously across hundreds of environments, the load on the central control plane grows non-linearly. The old monolithic setup turns into a single point of failure (SPOF). The background task queue accumulates, internal operation tracing becomes complicated, and update failures can disrupt neighboring processes. Safely managing large-scale infrastructures requires a dedicated architectural layer.
The Concept of an Orchestrator of Orchestrators
This specialized orchestration layer acts as an orchestrator of orchestrators. While standard Kubernetes manages containers within a single application, KubeTL coordinates the operation of multiple isolated clusters as a single ecosystem.
Implementing this architecture relies on three key engineering decisions:
-
Switching to a pull model for agents. Instead of sending commands from the central control plane to the client's private network, a dedicated component called
kubetl-agentis installed inside each cluster. This agent independently connects to the management servers via a secure gRPC connection to receive current tasks. This solution eliminates the need to open inbound ports in client networks, improves overall security, and resolves server accessibility issues behind NAT. - Job-Based Execution. Any action, whether installing updates, adding a new node, or deploying a cluster from scratch, is structured as a separate atomic job. This allows tracking the status of operations in real time, restarting them safely in case of network delays, and canceling specific tasks without the risk of disrupting neighboring environments.
- Managed Mass Rollouts. For global scheduled maintenance, we use task batching and rate limiting mechanisms. If an update group exceeds the allowable error threshold, the system automatically stops the process, which helps isolate the issue and prevents the failure from spreading across the entire infrastructure.
Engineering Standards and Results
This architectural approach allows overcoming the technological barriers of thousands of clusters while maintaining high service availability. Implementing the pull model and atomic jobs delivers predictable results:
- The number of internal platform incidents related to infrastructure management drops by approximately three times.
- The horizontal scaling process becomes fully controlled through proper capacity planning of the control plane resources.
Practical experience in executing large-scale IT transformations helps build reliable cloud products. Today, we are actively bringing these automation and security standards to UzCloud public services. The physical infrastructure of the platform is located in Uzbekistan, providing clients with minimal network latency and ensuring strict compliance with national data localization requirements.
If your IT team needs a reliable environment for working with containers, you can learn more about the technical capabilities and activate Managed Kubernetes in Uzbekistan.