Scaling and Load Balancing

Starting point

Every system begins the same way: one server running the application and the database together. It is simple, cheap, and completely adequate for a small user base.

Client → Server (app + database)

The first thing to break is usually resource contention — the database competes with the application for memory and CPU. So you split them:

Client → App Server → Database Server

Now each can be sized and tuned independently. This alone carries many systems a long way.

Vertical versus horizontal

When one server is no longer enough you have two options.

Vertical scaling means a bigger machine — more CPU, more RAM. It requires no code changes, which makes it attractive. But hardware has a ceiling, cost grows non-linearly, and you still have a single point of failure.

Horizontal scaling means more machines. It scales essentially without limit and gives you redundancy, but it forces you to solve distribution and shared state.

The usual path is to scale vertically until it gets expensive, then go horizontal.

Load balancers

Once you have several application servers, something must decide which one handles each request. That is the load balancer.

                    ┌→ App Server 1
Client → Load ──────┼→ App Server 2
         Balancer   └→ App Server 3

Beyond distribution it provides health checking (stop sending traffic to a dead server), SSL termination, and a single stable entry point.

Algorithms

• Round robin — rotate through servers in order. Simple; assumes servers and requests are uniform.
• Least connections — send to whichever server has the fewest active connections. Better when request durations vary.
• IP hash — hash the client IP to pick a server, so a given client consistently lands on the same one. Useful for session affinity, but distribution can become uneven.
• Weighted — give more traffic to more capable servers.

Round robin is the sensible default. Reach for least connections when some requests are much slower than others.

Layer 4 versus layer 7

A layer 4 balancer routes on IP and port. It is fast and protocol-agnostic but cannot see the request contents.

Statelessness is the prerequisite

Horizontal scaling only works if any server can handle any request. The moment a server keeps user data in local memory, load balancing breaks — the user's next request lands elsewhere and their session vanishes.

# Broken: state lives in one server's memory
sessions = {}

def login(user_id):
    sessions[user_id] = create_session()    # only this server knows

# Works: state lives in shared storage
def login(user_id):
    session = create_session()
    redis.set(f"session:{user_id}", session, ex=3600)

Push session state into a shared store such as Redis, or into a signed token held by the client. Then servers become interchangeable and you can add or remove them freely.

Sticky sessions (pinning a user to one server) are a workaround, not a solution — they undermine even distribution and lose data when that server dies.

Database scaling

Application servers are easy to replicate because they are stateless. Databases are the hard part, since they are state by definition.

Read replicas. Writes go to a primary; reads are served by copies. Since most systems read far more than they write, this helps immediately. The cost is replication lag — a read immediately after a write may return stale data.

              ┌→ Replica 1 (reads)
Primary ──────┼→ Replica 2 (reads)
(writes)      └→ Replica 3 (reads)

Sharding. Split the data itself across databases, each owning a subset of rows. This is the only way to scale writes, but it is genuinely difficult: cross-shard queries and transactions become expensive, and rebalancing is painful. Shard only when you must.

Common shard keys are user ID (keeps a user's data together) or a hash of the primary key (distributes evenly but destroys range queries).

Watch for hot spots

Even with perfect infrastructure, uneven access ruins throughput. If one celebrity account lives on one shard, that shard melts while others idle.

Mitigations include caching hot keys aggressively, splitting hot rows across several keys, and choosing a shard key with naturally even distribution.

A realistic progression

Do not build the final architecture on day one. Grow into it:

1. Single server
2. Separate application and database servers
3. Add a load balancer and multiple stateless application servers
4. Add a cache for hot reads
5. Add read replicas
6. Add a CDN for static assets
7. Shard the database
8. Split into services along team and domain boundaries

Each step should be justified by a real bottleneck you have measured. Skipping ahead adds operational cost with no benefit — and in an interview, jumping straight to step eight signals that you are reciting rather than reasoning.

Common mistakes

1. Sharding too early. It is the most expensive step and rarely the first one needed.
2. Keeping session state in server memory while claiming horizontal scalability.
3. Ignoring replication lag when a read must reflect a just-completed write.
4. Treating the load balancer as infallible — it needs redundancy too.
5. Forgetting the database is usually the bottleneck, not the application tier.

Summary

Scale vertically until it becomes uneconomical, then horizontally. Horizontal scaling requires stateless servers and a load balancer. Databases scale by read replicas first and sharding only when writes genuinely demand it. Add each layer in response to a measured bottleneck.