AKS at Scale: Hard-Won Lessons from 1000+ Node Clusters

AKS at Scale: Hard-Won Lessons from 1000+ Node Clusters

Nobody warns you about the point where Kubernetes stops behaving like Kubernetes. At 100 nodes the platform feels manageable: logs are searchable, deployments finish quickly, and most incidents resolve with a kubectl command and some patience. Cross 500 nodes and small architectural assumptions start cracking. Cross 1,000 nodes and those cracks become structural.

The problems described here are not hypothetical. etcd database sizes that stretched backup windows into hours. Observability stacks consuming more cluster resources than the workloads they were supposed to monitor. Network overlays running fine at 200 nodes that started dropping packets at 800. If you’re planning to push past 500 nodes, or already running infrastructure at that scale and things feel increasingly fragile, read on.

The Scale Cliff: Why 1,000 Nodes Changes Everything

At 100 nodes, Kubernetes feels manageable. Monitoring works. Logs are searchable. Network patterns make sense. Deployments complete in minutes. Then you cross 500 nodes and small cracks appear. By 1,000 nodes, those cracks become structural failures.

The problem: Kubernetes components designed for graceful degradation hit hard limits at scale. etcd performance degrades non-linearly with keyspace size. Network overlay solutions that worked fine at 200 nodes saturate at 800. Observability stacks consuming 3% of cluster resources at 100 nodes consume 25% at 1,000. Cost-per-node stays flat but operational overhead per node increases exponentially.

These aren’t bugs. They’re architectural realities. Understanding where the cliffs are lets you plan around them instead of discovering them in production outages.

etcd: The Hidden Scaling Bottleneck

etcd is the single most critical component in your cluster and the first to hit scaling limits. It stores all cluster state: every pod, service, config map, secret, and custom resource. At 1,000 nodes with 200 pods per node, you’re managing 200,000+ objects. etcd wasn’t designed for that scale without careful tuning.

Performance Degradation Patterns

etcd performance degrades based on keyspace size, transaction rate, and storage backend latency. At small scale, these factors don’t matter. At mega-cluster scale, they dominate operational behavior.

Symptoms you’ll see:

  • API server latency spikes during deployments
  • kubectl commands timing out intermittently
  • Controller reconciliation loops falling behind
  • Scheduler making suboptimal placement decisions due to stale state

The root cause is usually one of three things: etcd database size exceeding memory capacity, insufficient IOPS on the storage backend, or transaction rate overwhelming the commit pipeline.

Backup Size and Recovery Time

etcd backup size scales with keyspace. A 100-node cluster might produce 500MB backups. A 1,000-node cluster produces 8GB+ backups. That size creates operational problems:

  • Backup windows extend from minutes to hours
  • Network transfer costs increase linearly
  • Recovery time objectives (RTO) slip from “15 minutes” to “2+ hours”
  • Storage costs for retention policies multiply unexpectedly

Worse: most backup solutions for etcd aren’t tested at mega-cluster scale. The tooling that works reliably at 100 nodes silently fails or creates corrupted snapshots at 1,000 nodes.

Practical Mitigation

AKS manages etcd for you, but you still need to monitor and validate its health. Here’s a Terraform configuration that sets up Azure Monitor alerts for etcd-related API server latency:

resource "azurerm_monitor_metric_alert" "etcd_latency" {
  name                = "aks-etcd-high-latency"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_kubernetes_cluster.main.id]
  description         = "Alert when API server latency exceeds 200ms (etcd saturation signal)"
  severity            = 2
  frequency           = "PT1M"
  window_size         = "PT5M"

  criteria {
    metric_namespace = "Microsoft.ContainerService/managedClusters"
    metric_name      = "apiserver_request_duration_seconds"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 0.2

    dimension {
      name     = "verb"
      operator = "Include"
      values   = ["GET", "LIST", "PATCH", "UPDATE"]
    }
  }

  action {
    action_group_id = azurerm_monitor_action_group.platform.id
  }
}

resource "azurerm_monitor_metric_alert" "etcd_database_size" {
  name                = "aks-etcd-database-size-warning"
  resource_group_name = azurerm_resource_group.main.name
  scopes              = [azurerm_kubernetes_cluster.main.id]
  description         = "Alert when etcd database approaches size limits"
  severity            = 3
  frequency           = "PT5M"
  window_size         = "PT15M"

  criteria {
    metric_namespace = "Microsoft.ContainerService/managedClusters"
    metric_name      = "etcd_db_total_size_in_bytes"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 6442450944  # 6GB (warning threshold)
  }

  action {
    action_group_id = azurerm_monitor_action_group.platform.id
  }
}

These alerts won’t prevent etcd saturation, but they’ll give you advance warning before cascading failures occur. At scale, that early warning is the difference between a controlled maintenance window and an all-hands incident.

Network Performance: When Overlay Solutions Hit Limits

Network overlay performance is invisible at small scale and catastrophic at large scale. Container Network Interface (CNI) plugins that handle 50,000 pods without issue can saturate CPU and drop packets at 200,000 pods. There is no single right answer for which CNI to use. For a full breakdown of the tradeoffs between kubenet, Azure CNI, and Azure CNI Overlay, see AKS Networking Clash: kubenet vs. CNI vs. CNI Overlay.

Pod Density and Node Saturation

Azure CNI Overlay supports up to 250 pods per node. That’s a theoretical maximum. Practical limits depend on network I/O patterns, pod churn rate, and service mesh overhead.

Signals that you’re approaching saturation:

  • Nodes showing high system CPU (kernel networking overhead)
  • Intermittent packet loss between pods on the same node
  • Service discovery latency increasing over time
  • DNS resolution failures under load

The underlying issue: network namespace creation, iptables rule updates, and conntrack table management all scale poorly. At 200 pods per node, these operations consume negligible resources. At 250 pods per node, they dominate system CPU.

Cross-Node Latency Patterns

Overlay networks add encapsulation overhead. Azure CNI Overlay typically adds 100-200 microseconds per hop. At small scale, that’s noise. At mega-cluster scale, it compounds across multi-tier applications.

Example: a request traversing frontend → API gateway → backend service → database proxy touches 4 pods. If those pods span nodes, you’ve added 400-800 microseconds of latency from network overhead alone. Multiply that by 10,000 requests per second and the impact becomes measurable in user-facing metrics.

Mitigation Strategy

  • Pin latency-sensitive workloads to the same node using pod affinity
  • Use host networking for data-plane components (with appropriate security controls)
  • Monitor conntrack table utilization: sysctl net.netfilter.nf_conntrack_count
  • Set conservative pod density limits (180-200 pods/node instead of 250)
  • Implement service mesh with extended Berkeley Packet Filter (eBPF) dataplane (Cilium) to reduce iptables overhead

These aren’t performance optimizations. They’re operational requirements at scale.

Observability Overhead: When Monitoring Becomes the Problem

Observability at scale creates a paradox: the systems you need to diagnose problems become the source of resource exhaustion.

Logging Cost Explosion

A single pod generating 100KB/day of logs costs nothing. 200,000 pods generating the same logs produce 20GB/day. Over a month, that’s 600GB. With 3x replication and 90-day retention, you’re storing 162TB of log data.

Storage costs for that volume run into thousands of dollars monthly. Query performance degrades. Log ingestion pipelines fall behind. The tooling designed to help you debug problems becomes unusable during incidents.

Metric Cardinality Problems

Prometheus-based monitoring hits cardinality limits around 10 million active time series. A 1,000-node cluster with moderate instrumentation easily exceeds that threshold:

  • 200,000 pods × 20 metrics per pod = 4M series
  • 1,000 nodes × 100 metrics per node = 100K series
  • 50 services × 10K instances × 5 metrics = 2.5M series
  • Custom application metrics add another 3M+ series

When you exceed cardinality limits, Prometheus becomes unstable. Queries time out. Dashboards fail to render. Alerting rules stop evaluating. You lose observability exactly when you need it most.

Practical Approaches

  • Implement aggressive log sampling: 1% sampling still gives 2GB/day of logs
  • Use structured logging with consistent field names to enable efficient compression
  • Archive cold logs to blob storage (pennies per GB vs. dollars per GB in hot storage)
  • Deploy federated Prometheus with careful metric filtering at scrape time
  • Use recording rules to pre-aggregate high-cardinality metrics
  • Consider managed observability services (Azure Monitor, Datadog) that handle scale for you

The honest assessment: if your observability stack consumes more than 10% of cluster resources, it’s time to rethink your approach. At mega-cluster scale, that threshold is easy to exceed.

Cost Spirals: Small Decisions with Exponential Consequences

Cost optimization at 100 nodes is optional. At 1,000 nodes, it’s mandatory. Small inefficiencies compound brutally.

Resource Overprovisioning

Teams typically request 2x actual resource needs for safety margin. At 100 nodes, that’s wasteful but affordable. At 1,000 nodes with 250 pods per node, you’re paying for 125,000 unutilized CPU cores.

With Azure D8s_v5 nodes at ~$0.40/hour, a 1,000-node cluster costs ~$288,000/year in compute alone. 50% overprovisioning adds $144,000 annually. That’s real budget impact.

Storage Cost Patterns

Every pod gets ephemeral storage. Most clusters also provision persistent volumes. At scale, storage costs exceed compute costs.

Example: 200,000 pods with 10GB ephemeral storage each = 2PB of ephemeral storage. Persistent volume claims add another 500TB+. Azure Premium SSD costs $0.135/GB/month. You’re paying $300K+ monthly for storage alone.

Network Egress Surprises

Cross-region and internet egress costs scale linearly with traffic volume. A 1,000-node cluster handling 10TB/day of egress traffic incurs $1,500/day in bandwidth costs ($45,000/month).

Teams typically discover these costs 60 days into a scale-up when the first full billing cycle completes. By then, architectural changes are expensive and disruptive.

Cost Control Strategy

  • Implement cluster autoscaling with aggressive scale-down policies
  • Use spot instances for fault-tolerant workloads (70% cost reduction)
  • Right-size pod resource requests using VPA (Vertical Pod Autoscaler)
  • Enable Azure Hybrid Benefit for Windows nodes
  • Deploy regional caching layers to reduce cross-region egress
  • Monitor and alert on cost metrics, not just resource metrics

Teams defer cost optimization in favor of operational simplicity, and early on that is usually the right call. At mega-cluster scale, that priority reverses. Cost efficiency becomes a constraint you cannot ignore. AKS Cost Optimization: Resource Governance That Actually Works goes deeper on VPA configuration and autoscaling policies if you want the practical implementation details.

Debugging at Scale: Finding Needles in Exponentially Larger Haystacks

Debugging a 100-node cluster means checking logs from a few thousand pods. Debugging a 1,000-node cluster means isolating the problem from millions of log lines across 200,000+ pods.

Correlation and Isolation

When a user reports an error, your troubleshooting workflow looks like this:

  1. Identify the service handling the request (1 of 50+ services)
  2. Find the pod instance that processed the request (1 of 5,000+ pod instances)
  3. Locate the relevant log lines (1 of 10M+ log events in the time window)
  4. Correlate with upstream/downstream service calls
  5. Reproduce the issue in a controlled environment

At small scale, steps 2-3 take minutes. At mega-cluster scale, they take hours, assuming correlation IDs exist and work correctly. Without proper instrumentation, they’re impossible.

Reproduction Challenges

Issues that reproduce reliably at scale rarely reproduce in test environments. A race condition that triggers once per 100,000 requests never manifests in pre-production. Network congestion patterns that emerge at 1,000 nodes don’t exist at 10 nodes.

This creates a diagnostic blind spot. You can observe the failure in production but can’t reproduce it for root cause analysis.

Large-Scale Troubleshooting Checklist

Here’s a diagnostic script I use for investigating performance degradation at scale:

#!/bin/bash
# Large-scale AKS cluster diagnostic script
# Run this when experiencing unexplained performance issues

set -euo pipefail

CLUSTER_NAME="${1:?Cluster name required}"
RESOURCE_GROUP="${2:?Resource group required}"
OUTPUT_DIR="./diagnostics-$(date +%Y%m%d-%H%M%S)"

echo "Running diagnostics for cluster: $CLUSTER_NAME"
mkdir -p "$OUTPUT_DIR"

# Get cluster credentials
az aks get-credentials --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --overwrite-existing

# Node health check
echo "Checking node health..."
kubectl get nodes -o wide > "$OUTPUT_DIR/nodes.txt"
kubectl top nodes > "$OUTPUT_DIR/node-resources.txt" || echo "Metrics server unavailable" > "$OUTPUT_DIR/node-resources.txt"

# API server latency check
echo "Checking API server latency..."
for i in {1..5}; do
  time kubectl get nodes > /dev/null 2>&1
done 2>&1 | grep real > "$OUTPUT_DIR/api-latency.txt"

# etcd health indicators
echo "Checking etcd health signals..."
kubectl get --raw /metrics | grep -E "apiserver_request_duration|etcd_request_duration" > "$OUTPUT_DIR/etcd-metrics.txt" || echo "Metrics unavailable" > "$OUTPUT_DIR/etcd-metrics.txt"

# Pod distribution analysis
echo "Analyzing pod distribution..."
kubectl get pods -A -o json | jq -r '.items[] | "\(.spec.nodeName)"' | sort | uniq -c | sort -rn > "$OUTPUT_DIR/pod-distribution.txt"

# Network policy count (can cause iptables overhead)
echo "Checking network policy count..."
kubectl get networkpolicies -A --no-headers | wc -l > "$OUTPUT_DIR/netpol-count.txt"

# Service endpoint count (affects kube-proxy performance)
echo "Checking service endpoint count..."
kubectl get endpoints -A -o json | jq '[.items[].subsets[].addresses] | flatten | length' > "$OUTPUT_DIR/endpoint-count.txt"

# Resource pressure signals
echo "Identifying pods with resource pressure..."
kubectl get pods -A -o json | jq -r '.items[] | select(.status.conditions[]? | select(.type=="Ready" and .status=="False")) | "\(.metadata.namespace)/\(.metadata.name)"' > "$OUTPUT_DIR/not-ready-pods.txt"

# Recent events (truncated for performance)
echo "Capturing recent cluster events..."
kubectl get events -A --sort-by='.lastTimestamp' | tail -1000 > "$OUTPUT_DIR/recent-events.txt"

# Node condition checks
echo "Checking for node pressure conditions..."
kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[]? | select(.type=="MemoryPressure" or .type=="DiskPressure" or .type=="PIDPressure") | select(.status=="True")) | .metadata.name' > "$OUTPUT_DIR/nodes-under-pressure.txt"

# ConfigMap and Secret count (affects etcd size)
echo "Counting ConfigMaps and Secrets..."
echo "ConfigMaps: $(kubectl get configmaps -A --no-headers | wc -l)" > "$OUTPUT_DIR/object-counts.txt"
echo "Secrets: $(kubectl get secrets -A --no-headers | wc -l)" >> "$OUTPUT_DIR/object-counts.txt"
echo "Total Pods: $(kubectl get pods -A --no-headers | wc -l)" >> "$OUTPUT_DIR/object-counts.txt"

# DNS performance check
echo "Testing DNS resolution performance..."
kubectl run dns-test --image=busybox:1.36 --restart=Never --rm -i --command -- sh -c "time nslookup kubernetes.default" > "$OUTPUT_DIR/dns-test.txt" 2>&1 || echo "DNS test failed" > "$OUTPUT_DIR/dns-test.txt"

echo "Diagnostics complete. Results in: $OUTPUT_DIR"
echo ""
echo "Quick analysis:"
echo "Nodes: $(kubectl get nodes --no-headers | wc -l)"
echo "Pods: $(kubectl get pods -A --no-headers | wc -l)"
echo "Not Ready Pods: $(cat $OUTPUT_DIR/not-ready-pods.txt | wc -l)"
echo "Nodes Under Pressure: $(cat $OUTPUT_DIR/nodes-under-pressure.txt | wc -l)"
echo ""
echo "Review the output files for detailed diagnostics."

This script collects the signals that matter at scale: API latency, pod distribution skew, resource pressure indicators, and object count metrics. It doesn’t solve problems, but it eliminates 90% of the noise.

Patterns That Prevent Catastrophe

After running mega-clusters through multiple incident cycles, a few patterns consistently prevent the worst outcomes:

Progressive rollouts: Never deploy to 1,000 nodes simultaneously. Deploy to 1 node, then 10, then 100, then all. Automate rollback triggers. This pattern catches 95% of scale-dependent bugs before they impact production.

Blast radius isolation: Segment your cluster into failure domains using node pools, namespaces, and network policies. When something fails (and it will), contain the damage. AKS Network Policies: The Security Layer Your Cluster Is Missing covers practical policy configuration if you are starting from scratch.

Capacity reservation: Reserve 15-20% headroom for burst traffic and incident response. Running at 90%+ utilization saves money until you need to scale during an outage and can’t.

Immutable infrastructure: Treat nodes as cattle, not pets. Automate node replacement on a fixed schedule (weekly or monthly). This prevents subtle configuration drift that compounds into unreproducible failures.

Operational runbooks: Document every common failure mode. When API server latency spikes at 2 AM, you don’t want to be reading Kubernetes source code to understand etcd compaction behavior.

These patterns aren’t revolutionary. They’re boring, defensive engineering. At mega-cluster scale, boring wins.

Honest Takeaways

Running AKS at 1,000+ nodes isn’t fundamentally different from running it at 100 nodes. It’s exponentially different. Problems that self-heal at small scale cascade catastrophically at large scale. Architectural decisions that feel premature at 50 nodes become load-bearing at 500 nodes.

If you’re planning to scale past 500 nodes: budget significant engineering time for operational tooling. Plan your observability strategy before your first node boots. Understand your cost model in detail. Test failure scenarios at scale before they happen in production.

If you’re already running at scale: you know everything in this article because you’ve lived it. The value isn’t the advice. It’s knowing you’re not alone in discovering these lessons the hard way.

Scale is honest. Every shortcut taken for velocity will surface eventually, usually at the worst possible moment. Budget engineering time to address that reality before you hit 500 nodes, not after. Fixing structural problems under production pressure costs significantly more than building them correctly from the start.

Comments