Load balancer: how to distribute traffic safely
How a load balancer distributes requests across backends for scale and high availability: layer 4 vs 7, algorithms, health checks and stateless sessions.
You spun up a second instance of your application because one couldn't handle the traffic anymore, and now you have two identical machines running the same code. The problem is obvious: how does the user know which one to send the request to? Pointing DNS straight at one IP works fine until that machine dies at three in the morning. And when you need a third instance, you start over. A load balancer exists to fix exactly this mess: it sits in front of your backends and decides, request by request, who serves it.
This article covers what a load balancer actually solves, the difference between operating at layer 4 and layer 7, the distribution algorithms, and why health checks and stateless sessions are the part that truly matters.
What a load balancer solves
A load balancer receives all the traffic hitting your domain and distributes it across a set of backends — what we usually call a pool or upstream. Two wins come for free. The first is horizontal scale: instead of buying a bigger machine (expensive and capped), you add identical instances and the LB starts sending them traffic. The second is high availability: if a backend dies, the LB stops sending requests to it and the user never notices.
Without a load balancer, both goals become manual, fragile work. With one, scaling is adding an address to the pool, and availability is a health check setting. The difference between an architecture that survives a traffic spike and one that falls over on the first holiday is usually that box in the middle.
Layer 4 versus layer 7
The most important configuration distinction is which layer of the network model the LB operates at.
A layer 4 load balancer works at the TCP/UDP level. It sees IP addresses and ports, and forwards packets without opening the contents. It's fast and cheap on CPU because it doesn't interpret anything passing through — it doesn't know whether there's HTTP, gRPC, or some binary protocol inside. Use layer 4 when you need raw throughput and don't need decisions based on request content.
A layer 7 load balancer works at the HTTP level. It reads the whole request: method, path, headers, cookies. That enables smart routing — sending /api/* to one pool and /static/* to another, or routing by a version header, or applying rules by hostname. The cost is that it has to parse HTTP and, usually, terminate TLS to see the content. In practice, most web applications want layer 7.
Layer 7 LB
/api/* ──────────────────────► pool-api (3 instances)
/static/* ──────────────────────► pool-static (2 instances)
/ ──────────────────────► pool-web (4 instances)
Distribution algorithms
Deciding which backend gets the next request is the job of the balancing algorithm. The three you'll find almost everywhere:
Round-robin distributes in sequence: first request to backend A, second to B, third to C, and back around. It's simple and works well when all instances have similar capacity and requests cost roughly the same. It's the default in most LBs for good reason.
Least-connections sends the next request to the backend currently holding the fewest active connections. It beats round-robin when requests have very different durations — one that takes 50ms and one that takes 8 seconds. With pure round-robin, one backend can pile up several slow requests while another sits idle; least-connections corrects that naturally.
IP hashing computes a hash of the source IP and always sends that client to the same backend. It creates affinity without cookies, but it has the same problem as sticky sessions, which I'll attack further down.
Health checks: the heart of availability
Distributing traffic is the easy part. The part that decides whether your high availability is real or theater is the health check. The LB tests each backend periodically and, when one fails, pulls that instance out of the pool. The following requests go to healthy backends, with no error reaching the user.
Here's the detail that separates a good health check from a useless one: what counts as "healthy." A TCP health check only verifies that the port accepts a connection — but a process can accept a connection and still have its database down, returning errors on everything. So prefer a layer 7 health check that hits a real endpoint, like /health, and judges by the HTTP status of the response. A load balancer's health check decides by the backend's HTTP status: a 200 keeps the instance in the pool, a 503 removes it. Make your /health endpoint actually check the dependencies (database, cache, queue) and return the right code — a lying 200 is worse than no health check, because it hides the problem.
health check:
path: /health
interval: 5s
timeout: 2s
healthy_threshold: 2 # 2 consecutive 200s → back in the pool
unhealthy_threshold: 3 # 3 consecutive failures → out of the pool
The thresholds exist to keep a momentary hiccup from knocking out a good instance. You want to remove a backend that genuinely died quickly, but you don't want to evict one that just had a half-second GC pause.
Sticky sessions and why to avoid them
A sticky session (or session affinity) is when the LB ties a user to a specific backend — usually via cookie — and sends all of that user's requests to the same instance. The classic motivation is that the application stores the user's session in that process's local memory. If the next request lands on another instance, the session "vanishes" and the user gets logged out.
It works, but it's a patch. Look at what sticky session breaks: when the backend the user is glued to goes down, their session goes with it, and the health check — meant to protect the user — now logs them out too. Balancing goes lopsided, because a node with many glued users piles up load while another sits idle. And deploys get painful, because taking down an instance means taking down the sessions stuck to it.
The opinion here is firm: make your instances stateless. Don't keep the session in the process's local memory. Put it in a shared store — Redis, a database, anything every instance sees equally. With sessions in a shared store, any backend serves any request from any user, and the LB is free to send traffic wherever makes the most sense. Sticky session isn't a feature you turn on to gain performance; it's a symptom that state is in the wrong place. Fix the cause, not the symptom.
TLS termination and failover
Two pieces complete the picture. TLS termination is the LB ending the HTTPS connection, decrypting the traffic, and speaking plain HTTP to the backends on the internal network. This centralizes certificates in one place (instead of spreading them across every instance) and frees backend CPU, which no longer has to do the TLS handshake. If the network between the LB and the backends isn't trusted, you can re-encrypt on the internal hop — but for most cases inside a VPC, terminating at the LB is the default.
Failover is the LB's behavior when backends become unavailable: it redirects traffic to the ones that remain. In more serious setups, this extends to failover across entire regions or availability zones. Worth remembering that a load balancer and a CDN solve different but complementary problems — if you're still fuzzy on that boundary, it's worth understanding what a CDN is before you draw the edge of your infrastructure.
Frequently asked questions
Are a load balancer and a reverse proxy the same thing?
They're overlapping but not identical concepts. A reverse proxy sits in front of servers and forwards requests; a load balancer is a reverse proxy specialized in distributing load across several backends. In practice, tools like Nginx and HAProxy do both roles, so the line is blurred. The difference is intent: you say "load balancer" when the focus is distributing and providing availability.
Where should SSL go: on the load balancer or on each backend?
In the vast majority of cases, on the load balancer, via TLS termination. You centralize certificate management, simplify renewal, and take the handshake CPU cost off the backends. Terminating TLS on each backend only makes sense if regulation or topology demands end-to-end encryption — and even then, you usually re-encrypt between the LB and the backend rather than giving up the LB.
How many backends do I need for high availability?
At least two, in different zones or hosts. With a single instance, the load balancer has nowhere to send traffic when it goes down — it just relays the error. Two is the floor for a backend failure to be absorbed; three or more give you room to also pull an instance for deploys without losing redundancy.
Is sticky session always bad?
Not in absolute terms, but it's almost always a sign of an architecture problem. If you need sticky session because you store the session in local memory, the right fix is moving the session to a shared store, not tying the user to a node. There are legitimate, rare cases — protocols with long connection state, for example — but for an ordinary web app, treat sticky session as technical debt.
What to take away
A load balancer isn't just a box that splits requests; it's the piece that turns "I have a server" into "I have a system that scales and doesn't fall over." Start by choosing the right layer — layer 7 for most web applications — and a simple algorithm like round-robin or least-connections. Invest seriously in the health check, because that's what makes availability real: point it at an endpoint that checks dependencies and judges by HTTP status. And above all, make your instances stateless. When any backend can serve any request, the load balancer does its job properly, deploys stop hurting, and you never again need a sticky session to hide misplaced state.
- 01 Does the PageSpeed Score Affect SEO? 100/100 and Failing Google doesn't rank on PageSpeed's 0 to 100 score but on Core Web Vitals from real users. Here's why a site scores 100/100 and still fails in the field.
- 02 Unit test vs integration test: the difference in practice The difference between unit and integration tests with API and database examples: what each one catches, what it costs, and when to use each in your project.