A latency budget is a total response-time target that you divide up and assign to every component in the request path.

You create one because latency accumulates silently. Each team adds "just 50 ms" for something reasonable, and the page ends up taking two seconds with nobody responsible.

A budget flips this around. Instead of measuring at the end and hoping, you allocate up front — so if the ranking service exceeds its 50 ms slice, that's a specific bug with a specific owner rather than a vague complaint that the site feels slow.

Latency is how long a single request takes. Throughput is how many requests complete per unit of time.

Using a highway analogy: latency is how long your car takes to reach the destination, throughput is how many cars per hour the highway carries, and bandwidth is the number of lanes.

They're independent, which surprises people. Adding servers increases throughput but does nothing for the latency of one request. And they can conflict — batching improves throughput but increases latency, because requests wait to be grouped.

Because averages hide the worst experiences. If nine requests take 10 ms and one takes 2,000 ms, the average is 209 ms — which sounds fine, but one user waited two full seconds.

p99 means 99% of requests are faster than that value, so it directly exposes the slow tail instead of smoothing it away.

There's a second reason that matters more than it seems: the slowest requests usually belong to users with the most data — the most followers, the biggest cart, the longest history. Those are your power users, so p99 isn't a rare edge case, it's the experience of the people who use your product most.

It's what happens when one request fans out to many services and you have to wait for all of them.

If a request calls 100 services in parallel and each has a p99 of 100 ms — meaning each is slow just 1% of the time — the probability that all 100 are fast is 0.99¹⁰⁰ ≈ 0.37.

So 63% of user requests hit at least one slow call, even though every individual service is fast 99% of the time.

This is why big microservice systems feel sluggish while every service dashboard looks healthy. The fixes are reducing fan-out, hedged or tied requests, and hard timeouts with fallback values.

First state the target as a percentile: p99 under 300 ms.

Then list every hop and assign a slice — roughly 40 ms for the client-to-edge network, 10 ms for TLS and the load balancer, 15 ms for auth, 5 ms for a Redis lookup, 80 ms for the search query, 50 ms for ranking, 20 ms for serialization, and 40 ms for the network back.

That totals 260 ms, leaving about 40 ms of headroom for GC pauses and retries.

Two things stand out. The network is 80 ms — over a quarter of the budget — and no code change fixes that; only a CDN or a closer region does. And auth, search, and ranking may be parallelizable, which would cut the server-side portion significantly.

The relative scale matters more than the exact figures. The key anchors are: RAM access around 100 nanoseconds, a random SSD read around 150 microseconds, a round trip within the same data center around 0.5 milliseconds, a spinning-disk seek around 10 milliseconds, and a cross-continent round trip around 150 milliseconds.

The useful way to hold this is a human-scale analogy. If an L1 cache read took one second, RAM would take three minutes, an SSD read about two days, a same-datacenter hop about eleven days, and a California-to-Netherlands round trip about ten years.

That's why caching is so effective — you're skipping the ten-year trip. And it's why cross-region latency can't be engineered away: it's bounded by the speed of light, so the only fix is moving data closer to users.

Sequential calls add up; parallel calls cost only as much as the slowest one.

If auth takes 15 ms, search takes 80 ms, and ranking takes 50 ms, running them in sequence costs 145 ms. Running them in parallel costs 80 ms — the maximum rather than the sum.

This is usually the cheapest latency win available, because it needs no extra infrastructure, just restructured code. The constraint is data dependencies: if ranking needs the search results, it genuinely has to wait. But a surprising number of calls in real codebases are sequential purely out of habit.

A hedged request means sending the same request to two replicas, taking whichever answer arrives first, and cancelling the other.

It works because slowness is usually caused by transient local conditions — a GC pause, a busy disk, a noisy neighbour — and it's unlikely two replicas hit that at the same moment. So hedging cuts the tail dramatically.

The cost is roughly double the load on that call. A common refinement is to hedge only after waiting for the p95 — so you only pay the extra cost on the 5% of requests that already look slow.

Tied requests go further: both replicas are told about each other, so whichever starts first cancels its twin, which gives most of the benefit with far less waste.

An SLI is the indicator — the thing you actually measure, such as the p99 response time of GET /search.

An SLO is the objective — your internal target for that measurement, such as p99 under 300 ms for 99.9% of the month.

An SLA is the agreement — a contractual promise to a customer, usually with financial penalties, such as p99 under 500 ms or they receive service credits.

The important practice is setting the SLO stricter than the SLA, often at about half the threshold. That gap is your safety margin: you get alerted and can fix the problem before you owe anyone money.

Because of queueing. As utilization approaches 100%, queueing delay grows toward infinity rather than linearly.

A server at 70% utilization has short queues and predictable latency. The same server at 95% has wildly worse latency even though it's only handling about 35% more traffic, because requests spend most of their time waiting rather than being processed.

This is why capacity planning and latency budgeting are really the same conversation. The practical rule is to run systems at 60–70% utilization so there's slack to absorb bursts, and to shed load or cap queue depth rather than letting queues grow unbounded.

They can quietly multiply it. A dependency with a 500 ms timeout and three retries isn't a 500 ms dependency — on the failure path it's a 1.5 second dependency.

That's especially dangerous because failure paths happen exactly when the system is already stressed, so retries add load at the worst possible moment.

The fixes are budgeting total elapsed time rather than just counting attempts, using exponential backoff with jitter so retries don't synchronize into a stampede, and adding circuit breakers so you stop retrying a dependency that's clearly down.

That gap says the slow path is structurally different from the typical path, not that everything is uniformly slow.

I'd start with distributed tracing, filtering to only the slow requests, and compare their span breakdown against fast ones to find which hop expands.

The usual suspects are cache misses falling through to a slow query, users with much larger data sets triggering unindexed or unbounded queries, GC pauses, lock contention, queueing during traffic spikes, and retries on a flaky dependency.

A useful discriminator is whether slow requests cluster in time or by user. Clustered in time suggests GC, deploys, or load spikes. Clustered by user suggests data-size problems like a missing index or an N+1 query on large accounts.