Median also lies more than 99th/100th percentiles because in many workloads the same client is making multiple parallel requests and often waiting for the slowest.<p>See this piece from Jeff Dean [0]<p>Quote:<p>"""
Component-Level Variability Amplified By Scale<p>A common technique for reducing latency in large-scale online services is to parallelize sub-operations across many different machines, where each sub-operation is co-located with its portion of a large dataset. Parallelization happens by fanning out a request from a root to a large number of leaf servers and merging responses via a request-distribution tree. These sub-operations must all complete within a strict deadline for the service to feel responsive.<p>Variability in the latency distribution of individual components is magnified at the service level; for example, consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure). Even for services with only one in 10,000 requests experiencing more than one-second latencies at the single-server level, a service with 2,000 such servers will see almost one in five user requests taking more than one second (marked “o” in the figure).<p>"""<p>[0] <a href="https://cacm.acm.org/research/the-tail-at-scale/#body-4" rel="nofollow">https://cacm.acm.org/research/the-tail-at-scale/#body-4</a>
It's little surprising that the author was doing perf work and not already comparing distributions. As the rest of the article outlines, you learn a lot more with more data!<p>It's complicated, but really worth learning how prometheus and grafana heatmaps combine if you want dashboards for real time service data. Multimodal distributions are basically the expected outcome given all the caching done in distributed systems.
Instead of the CDF I like to use one minus the CDF, the fraction of requests not yet completed at any given time. Then you can make it a log log plot showing the entire tail with details invisible in the CDF because in the CDF the tail is essentially a horizontal line at one.
An interesting statistical example if you haven’t seen it before is the Anscombe Quartet :<p><a href="https://en.wikipedia.org/wiki/Anscombe%27s_quartet" rel="nofollow">https://en.wikipedia.org/wiki/Anscombe%27s_quartet</a><p>Four sets of X,Y datapoints that have exactly (or very close) common statistical parameters (mean, variance, correlation, linear regression, R^2), but with vastly different spatial distributions and “behavior” when looked at visually.
I have mostly worked with computational physics with discontinuous polynomial approximations. Stuff like this would be easily detected in those methods as anatomy brcsuse of assumption pf smoothness.<p>Statistics on the other hand is more accepting of discontinuous data due to it's basis in measure spaces.<p>Hence In general, you should know what your data should look like before you aim to detect anamolies. But also, it might be a good practice with new automated research actors to always use both approximations (measure theory based and otherwise), to figure out what's going on.
With only having read the headline (as it is customary), I dare say: Aggregation without distribution always means nothing.
mean w/o std dev is just pointless, I tend to reject them altogether.<p>If I am to record something like queue times (as in latency), I'd have multiple samplings of both time enqueued and entries (or approx entries size), provided the queues have a reasonable size() method (e.g. not O(n) and not blocking). Even having max entries increased with lower overall latency, it'd speak of bursty behaviour.<p>In that regard - Recording percentiles en masse (production) is rather expensive when there are thousands of metrics. The avg/mean + max require few bytes (sub cacheline) and it can be implemented lock-free.