Skip to answer

What should a smart-glasses benchmark report besides average latency?

Report the sample count, median, slower-tail behavior and failure rate, along with the measurement definition. Say whether unsuccessful requests were excluded from timing. An average alone can hide a small number of long waits or a system that responds quickly only when it succeeds.

Keep failures in the story

Timing only completed requests can make an unreliable system look excellent. Report how many attempts completed and describe timeouts separately. For a small sample, the individual measurements can be more informative than a precise-looking percentile with little data behind it.

Publish enough of the distribution to show the ordinary experience and the frustrating cases. If a test contains a handful of runs, label it a small sample and avoid presenting its slowest observation as a stable prediction of future tail latency.

About this library

How we research these answers ↗

Back to the topic and practical exercise ↗