Kubernetes field guide · interactive edition

Autoscaling is a feedback loop, not a magic button.

Your HPA can make the right decision - and your system can still fall over. The missing idea is not configuration. It is time, signal quality, and the long path from desired replicas to useful throughput.

illustrative incident · 10:02:18
$ kubectl get hpa checkout-api
NAME           TARGETS     MINPODS   MAXPODS   REPLICAS
checkout-api   147%/65%    6         40        24

$ kubectl get pods | tail
checkout-api-7dc...   0/1   Pending
checkout-api-4fb...   0/1   ContainerCreating
checkout-api-1aa...   1/1   Running

p99 latency: 3.8s ↑   DB pool: 100% ↑

The replica count is a request for capacity.
It is not capacity itself.

An autoscaler observes the past, calculates a desired state, and asks the rest of the system to catch up. Every handoff adds delay. Every bottleneck can veto the expected gain.

A thermostat is useful here: it senses temperature after the room changes, activates equipment, waits for heat to move, then measures again. Turn it too aggressively and the room swings hot and cold. Kubernetes has more plumbing - and considerably more YAML.

01

Click through the control path

One loop. Ten places to wait.

Select a stage. The inspector shows the owner, the failure you will see, and the next useful question. The three bands separate workload decisions, infrastructure, and application readiness.

The return arrow matters: the system observes the result and repeats. HPA, node provisioners, and application behavior are related loops - not one identical algorithm.

02

The capacity chain

Three layers must agree.

A

Workload scaling

HPA changes a scalable target’s desired replicas from resource, custom, or external metrics. For CPU utilization, the target is relative to container CPU requests - not node capacity and not an absolute CPU percentage.

Roughly: desired replicas = current replicas × current metric ÷ target metric. The controller then applies tolerance, missing-metric handling, policies, and stabilization.

Kubernetes HPA docs ↗
B

Infrastructure scaling

If the scheduler cannot place new Pods, they remain Pending. Cluster Autoscaler adjusts predefined node groups based on unschedulable Pods; Karpenter directly creates NodeClaims and chooses suitable capacity from configured constraints.

Both still wait on cloud APIs, instance supply, bootstrap, registration, CNI, DaemonSets, and node readiness.

C

Application readiness

Running means containers started. Ready means readiness checks allow traffic. Neither guarantees linear throughput: warm caches, JITs, connection pools, partition ownership, and downstream limits decide what a replica can actually add.

A good readiness probe protects users from cold instances; a slow warm-up also lengthens the feedback loop.

Pod readiness docs ↗
desired replicas->scheduled Pods->ready endpoints->useful throughput->healthy dependencies
03

Interactive model

Explore the delay budget.

This is an intentionally simple planning model - not a benchmark. Change the assumptions to see whether enough capacity becomes ready before the latency budget expires.

The deliberately uncomfortable caveat: assumed per-replica throughput rarely stays constant. Contention, retries, locks, skewed load, and downstream saturation bend the line. Use load tests and saturation curves, not wishful multiplication.
04

The metric problem

You scale what you can see.

A resource metric describes consumption. A demand metric can describe work waiting to be served. Neither category is universally better; the useful signal is the one causally connected to scarce capacity and early enough to act on.

CPU math

Requests are the denominator.

If a Pod uses 500m CPU and requests 1000m, utilization is 50%. If it requests 250m, the same usage is 200%. Missing CPU requests can prevent HPA from calculating utilization for the affected container metric.

Averages hide shape

50% average can mean one hot Pod.

Uneven routing, sticky sessions, partition ownership, or one expensive tenant can leave a replica saturated while the fleet average looks calm. More replicas help only if work can reach them.

05

Failure modes

When the loop fights the system.

Delay -> overshoot01

What: demand stays high while capacity boots, so successive observations request still more replicas. When capacity arrives, it may exceed need.

Observe: desired replicas climbing while Ready changes slowly; a later burst of endpoints.

Improve: headroom, faster startup, sane scale-up policies, scheduled capacity. Trade-off: cost or slower reaction.

Noisy signal -> oscillation02

What: replicas repeatedly scale up and down around a threshold.

Observe: alternating recommendations and frequent Pod churn.

Improve: stabilization windows, tolerance, less noisy aggregation. Trade-off: slower convergence.

Aggressive scale-down -> relapse03

What: a brief dip removes warm capacity; returning demand finds the service cold again.

Observe: latency spikes shortly after downscale, repeated cold starts.

Improve: conservative scale-down, minimum replicas, Pod disruption controls. Trade-off: idle spend.

Bad requests -> bad decisions04

What: CPU targets become misleading and scheduling may pack Pods unrealistically.

Observe: throttling, OOMs, Pending Pods, or surprising CPU utilization percentages.

Improve: measure and right-size requests; consider VPA recommendations. Trade-off: migration effort and possible restarts.

More clients -> dependency collapse05

What: each new replica opens pools and adds concurrency against an already slow database or API.

Observe: app replica count rises while DB connections, lock waits, errors, and retries rise faster.

Improve: global concurrency limits, backpressure, caching, query work, load shedding. Trade-off: queued or rejected work.

Scale-to-zero -> cold first request06

What: no worker exists when demand returns. A trigger must be observed before capacity starts.

Observe: queue growth or delayed first response during activation.

Improve: keep one warm replica for latency-critical paths, buffer work safely. Trade-off: baseline cost.

HPA behavior supports separate scale-up/down policies and stabilization windows. The default downscale stabilization window is five minutes; controller flags can change cluster defaults. Official behavior reference ↗

06

Worked example · hypothetical

The seven-minute rescue.

A checkout API normally serves 600 req/s on six ready Pods. Load testing suggests about 120 req/s per Pod before latency bends. At 10:02, traffic jumps to 1,800 req/s.

required replicasceil(1,800 ÷ 120) = 15additional replicas15 − 6 = 9
Demand jumps

Existing theoretical capacity is ~720 req/s. Queues and latency grow immediately.

HPA observes

CPU crosses target; HPA raises desired replicas. This time is illustrative - the controller’s sync period is configurable.

Scheduler says “no”

Three Pods fit. Six remain Pending: insufficient CPU plus zone and topology constraints.

Provisioner reacts

Unschedulable Pods trigger infrastructure decisions. A valid instance must be available within quota and constraints.

Node joins

Bootstrap, networking, storage, and system DaemonSets complete. Pending Pods can finally bind.

Pods become Ready

Images pull, processes start, caches warm, readiness passes. Endpoints begin receiving traffic.

The bottleneck moves

Fifteen replicas each allow a 20-connection pool: 300 possible DB connections. The database limit is 220. Throughput flattens; timeouts and retries amplify load.

Change one assumption, change the incident.

07

Choose by workload shape

Different problems, different levers.

WorkloadSignal / strategy that often fitsWhere it can fail
CPU-bound HTTP APIHPA on CPU with credible requests; modest warm headroomUneven routing, throttling, or startup slower than spike
I/O-heavy APIConcurrency, request rate plus saturation guardrails; explicit admission controlCPU stays low while database waits and latency rise
Queue workerKEDA on queue depth/age; derive replicas from drain-rate targetPoison messages, partition limits, scale-to-zero activation delay
Scheduled batchScheduled capacity, Jobs, queueing; resource isolationAutoscaler reacts after the known burst; batch starves services
Slow-start serviceMinimum warm pool, scheduled/predictive scaling, image and init optimizationController is faster than actuation; overshoot and costly idle replicas

HPAchanges workload replica targets

VPArecommends or changes Pod resource sizing

CA / Karpentercreate or remove node capacity

KEDAconnects event sources to workload scaling, including zero where supported

These tools solve different control problems. VPA may evict or recreate Pods depending on mode and platform support; combining VPA and HPA on the same CPU or memory signal requires care. VPA project ↗ · KEDA scaling concepts ↗

08

Incident mode

Find where capacity stopped.

autoscaling triage
Targets, replica counts, namespaces
Conditions, metrics, events, policies
Placement, readiness, restarts
Scheduling and probe failures
Chronological cluster clues

Recent resource usage - not scheduling capacity or demand

Pending? Read scheduler events and provisioner logs/events. Check requests, taints, affinities, topology, volumes, quotas, instance availability, and fragmentation.

Ready but still slow? Leave Kubernetes for a moment. Inspect load distribution, concurrency, pool limits, downstream latency, errors, retries, locks, queue age, and real completed work.

09

Design review checklist

Engineer the loop, not the YAML.

Autoscaling cannot manufacture database connections, cloud quota, partition parallelism, or time. It can only steer the capacity mechanisms you already engineered.

The best autoscaling system is not the one that moves fastest.
It is the one whose signal, delay, and limits match reality.

Research notes

Sources & claim boundaries

Behavior claims link to canonical project documentation. Timings, traffic, capacity, and database limits in the worked example are explicitly hypothetical. Recommendations are engineering interpretation, not Kubernetes guarantees.