Your dashboard says average response time is a healthy 120ms. Management is happy. And yet the complaints keep coming: "it's so slow sometimes." Are the users imagining it?
They're not. The average is hiding the truth — and the tool that reveals it is percentiles. Once you start reading p95 and p99 instead of averages, you finally see your service the way your slowest (and angriest) users do.
Why averages lie
An average smooths everything into one number, which means a few very slow requests vanish into a sea of fast ones. Consider ten requests, nine of them fast and one terrible:
50, 55, 60, 50, 65, 55, 60, 50, 55, 3000 (ms)
The average is ~350ms — already misleading. But nine of those users had a great time and one waited three seconds. The average describes nobody's actual experience: it's worse than the happy nine and far better than the miserable one.
Averages answer "what's typical?" — but reliability is about the worst experiences, not the typical one. The user who waited 3 seconds is the one who tweets about it.
What percentiles actually mean
A percentile tells you the value below which a given share of requests fall:
| Percentile | Means |
|---|---|
| p50 (median) | Half of requests were faster than this |
| p95 | 95% were faster — the slowest 1-in-20 |
| p99 | 99% were faster — the slowest 1-in-100 |
So "p99 latency = 800ms" means 99% of requests were quicker than 800ms, and the slowest 1% were worse. That slowest 1% is exactly where the pain lives.
- p50 tells you the typical experience.
- p95 / p99 tell you the bad experience — the long tail.
Why the tail matters more than you think
"Only 1% of requests are slow — who cares?" Here's the trap: a single user makes many requests. Load one page and you might fire 50 API calls. If 1% of calls are slow (p99), the chance that at least one of those 50 is slow is high — so a "1% problem" hits a large share of actual page loads.
The more requests a user's experience depends on, the more your tail latency (p95–p99.9) becomes their typical experience. At scale, the tail is the product.
p50 vs p99 in practice
| Metric | What it tells you | Watch it for |
|---|---|---|
| p50 (median) | The typical request | General health, baselines |
| p95 | The slow-ish minority | Early warning of degradation |
| p99 | The genuinely painful tail | The experience that drives complaints |
| Average | …honestly, mostly skip it | It hides both the typical and the tail |
A healthy practice: set your alerts and SLOs on a percentile, not the average. "p95 under 300ms" is a promise about real experience; "average under 300ms" can be true while a chunk of users suffer.
A quick gut-check
- If someone quotes you an average latency, ask for the p95/p99. That's where the story is.
- A growing gap between p50 and p99 means your tail is getting worse — often the first sign of trouble (an overloaded resource, a slow dependency) before the median moves at all.
The bottom line
| Idea | In one line |
|---|---|
| Averages hide outliers | They describe nobody's real experience. |
| Percentiles reveal the tail | p95/p99 show the slow, painful requests. |
| The tail compounds | One user hits many requests — 1% slow adds up fast. |
| Measure on percentiles | Set alerts and SLOs on p95/p99, not the mean. |
Stop asking "what's our average response time?" and start asking "what's our p99?" It's a small change in vocabulary that completely changes what you can see — and it's the difference between a dashboard that looks healthy and one that tells the truth.
This thinking carries straight into incident metrics (averages hide bad incidents too) and response-time monitoring.