Measuring Tail Latency Without Lying to Yourself
Every latency dashboard shows a p99. Almost every p99 is computed in a way that makes it meaningless. The problem is not the percentile; it is what gets averaged before the percentile is taken.
The aggregation trap
Suppose you have ten servers, each handling a thousand requests per minute. Each server computes its own p99 and reports it. The dashboard averages the ten p99 values. This is not the p99 of the system. It is the mean of ten different p99s, which is a statistic with no defined meaning and no useful interpretation.
The correct approach is to ship histograms, not percentiles, and merge the histograms before computing the percentile. Merging percentiles is not possible; merging histograms is trivial. Any metric system that does not support histogram aggregation cannot give you a correct tail latency number, regardless of how many decimal places it prints.
Coordinated omission
This is the other classic error, and it is subtler. If your load generator waits for a response before sending the next request, and a response takes ten seconds, then your generator only sends one request during those ten seconds. The slow request is counted once, and the nine seconds it blocked are invisible. Your measurement says the system handled one slow request; the system actually failed to serve many requests during that window.
The fix is to schedule requests on a fixed timeline, independent of responses, and record the difference between when a request was supposed to be sent and when it actually was. That difference is coordinated omission, and it is often larger than the latency you thought you were measuring.
What good looks like
- Histograms end to end, merged server-side, never averaged
- Load generators with open-loop scheduling and a correction term
- Separate measurement of queue time and service time
- Explicit recording of timeouts as a distinct outcome, not as a latency
The last one is underrated. A timeout is not a slow request; it is a request that never completed. Folding timeouts into the latency distribution truncates the tail and makes the system look better than it is.