Auto scaling: when to scale horizontally

Black Friday kicked off and the team had set autoscaling to spin up replicas when CPU crossed 70%. It worked — the cluster inflated to 40 pods. The problem: latency stayed terrible. CPU was low because requests were all stuck waiting on a slow database query, and the database was a single instance. Scaling horizontally multiplied consumers of a bottleneck that didn't scale. Autoscaling doesn't fix bad architecture; it amplifies what you already have, for better and for worse.

This article is about deciding when scaling horizontally makes sense and when it's just burning money. If your plan is to spread traffic across multiple replicas, also read how a load balancer distributes requests safely — autoscaling without a healthy balancer in front delivers nothing.

Vertical vs horizontal: not the same decision

Scaling vertically means swapping the machine for a bigger one: more CPU, more RAM. Simple, no code change, but it has a physical ceiling and each upgrade usually requires a restart (downtime). Scaling horizontally means adding more instances of the same application behind a balancer. No practical ceiling, but it requires the application to tolerate running as N copies at once.

The mental rule I use: vertical solves "my machine is too small," horizontal solves "I need more machines." Different problems. Plenty of people reach for horizontal when what they needed was to swap an undersized instance for a bigger one — and then carry coordination complexity for no reason.

What autoscaling actually solves

Autoscaling solves variable load: traffic that rises and falls through the day, predictable spikes (campaigns) and unpredictable ones (going viral). You pay for capacity close to real demand instead of provisioning for the peak around the clock.

What autoscaling does not solve:

• A slow query. More replicas = more connections hitting the same bottlenecked database. You scale the problem.
• A memory leak. New replicas leak just the same; you only delay the OOM and pay more for it.
• Deadlock or lock contention. More consumers fighting over the same lock makes it worse.

If the metric triggering the scaling is a symptom of a single-point bottleneck, autoscaling will make it worse, not better. Diagnose before you scale.

Statelessness is a prerequisite, not a detail

For an application to scale horizontally without bizarre bugs, every instance must be interchangeable. That means: no session state in process memory, no files written to local disk that another replica needs to read, no assumption of "always lands on the same pod."

So where does state live:

• Sessions → Redis, or a stateless JWT.
• Uploads/files → object storage (S3 and friends), never the instance's local disk.
• Cache → a shared store, not process memory (or a local cache that tolerates inconsistency).

If you turn on autoscaling for a stateful application, the classic symptom is the user getting logged out on every request because the balancer threw them at a pod that doesn't have their session. No hacked-in sticky sessions — fix the state at the source.

Which metric to scale on (CPU is a trap)

CPU is every tutorial's default and almost always the wrong metric. An I/O-bound application — which is most web apps — spends its time waiting on the database and external APIs, with low CPU while latency explodes. Scaling on CPU in that case triggers too late, or never.

Better metrics, in order of preference:

• Request latency (p95/p99): it's what the user feels. If p95 crosses the SLA, scale.
• Queue depth: for workers, queue size is the most honest signal of "I can't keep up."
• Requests per second per instance: if you know how many RPS one replica handles with headroom, scaling on RPS is predictable.
• CPU: valid only for genuinely CPU-bound work (image processing, compression, crypto).

Scale-up and scale-down aren't symmetric

Adding a replica should be fast and aggressive: the traffic is already here, delay costs SLA. Removing one should be slow and conservative. If you drop replicas at the first lull, you get flapping — up, down, up, down — and each cycle pays a cold start, kills connections, and trashes the cache.

That's why cooldowns are asymmetric:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0      # scale up immediately
    policies:
      - type: Pods
        value: 4
        periodSeconds: 30
  scaleDown:
    stabilizationWindowSeconds: 300    # wait 5 min of calm before scaling down
    policies:
      - type: Pods
        value: 1
        periodSeconds: 60

Scale up fast, scale down slow. The cost of one extra replica for a few minutes is trivial next to the cost of flapping.

Cost traps and when NOT to autoscale

Autoscaling without a ceiling (maxReplicas) is a blank check: a botched retry loop or an attack can inflate the cluster until the bill is scary. Always set a max. Another trap: scaling the application but not the database — you multiply connections and blow through the Postgres pool, taking everything down.

When not to autoscale:

• Stable, predictable load. Fixed capacity is simpler and cheaper; autoscaling adds complexity with no gain.
• The database as the real bottleneck. Fix the query, add an index, add a cache. Scaling the app doesn't touch the problem.
• A stateful application you can't (yet) make stateless. Fix the state first.

What to decide before turning autoscaling on

Before enabling autoscaling, answer three things: is the application truly stateless? Does the scaling metric reflect user pain (latency/queue) rather than a misleading proxy like CPU under I/O-bound load? And is the bottleneck actually in the layer you're about to scale — or in the database underneath it? Configure aggressive scale-up, conservative scale-down with a cooldown, and a maxReplicas that protects the bill. Autoscaling is an excellent tool for variable load and an expensive way to mask architecture problems. Knowing the difference is the whole job.

Frequently asked questions

Should I scale on CPU or latency?

Latency (p95/p99) for the vast majority of web apps, because they're I/O-bound and sit waiting on the database and APIs with low CPU. CPU only makes sense for genuinely CPU-bound work, like image processing or compression. Scaling on CPU in an I/O-bound app triggers late or never.

Does autoscaling fix a slow query?

No, it makes it worse. More replicas means more connections hitting the same bottlenecked database, and you scale the problem instead of the solution. Fix the query, add an index or a cache before thinking about scaling the application.

Why do my replicas keep going up and down all the time?

That's flapping, caused by symmetric or absent cooldowns. Configure fast scale-up and slow scale-down, with a stabilization window of several minutes before removing replicas, so you don't react to every momentary swing in load.

Do I need a load balancer to do autoscaling?

Yes. Replicas only receive traffic if something in front distributes requests across them. Without a healthy load balancer and correct health checks, spinning up replicas delivers no capacity to the user at all.