LLM inference performance involves multiple metrics beyond a single latency number. Key metrics include latency (total request time), time to first token (TTFT), time per output token (TPOT), throughput (tokens or requests per second), memory usage, utilization, and cost per token. These metrics serve different stakeholders: users care about response time, operators focus on hardware efficiency and cost, and researchers need to ensure optimizations don't degrade output quality.
Two requests with identical token counts can have vastly different performance profiles. Request A with 2,000 prompt tokens and 20 output tokens stresses the prefill phase, while Request B with 20 prompt tokens and 2,000 output tokens stresses the decode phase. This distinction matters for benchmarking, requiring developers to record prompt and output tokens separately.
Tail latency is critical because users notice when some requests take much longer than average. Beyond mean or median latency, practitioners should report high-percentile latencies such as p90, p95, and p99 to capture worst-case performance. A Python function using NumPy can compute these percentiles from a list of latency values, providing a comprehensive view of system performance.