mlsanityData to 7 Oct 2026

Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

arXiv 2407.07000 ↗First posted 9 Jul 2024cs.LG cs.AI cs.CLcited by 9

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 22 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

11 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 9

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  4. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  6. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  7. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  8. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  9. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 6.

9 papers.

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  2. 2024Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Studycited by 7
  3. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  4. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Acceleratorcited by 3
  5. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  6. 2025iServe: An Intent-based Serving System for LLMscited by 1
  7. 2025AI Compute Architecture and Evolution Trendscited by 1
  8. 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Studycited by 1
  9. 2024Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Evaluating Performance of LLM Inference Serving Systems
  2. 2023Efficient LLM Inference on CPUs
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  4. 2025Can abstract concepts from LLM improve SLM performance?
  5. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
  6. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  7. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  8. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments