mlsanityData to 14 Oct 2026

Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving

arXiv 2410.14257 ↗First posted 18 Oct 2024cs.LG cs.AIcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

User experience is a critical factor Large Language Model (LLM) serving systems must consider, where service level objectives (SLOs) considering the experience of individual requests and system level metrics (SLMs) considering the overall system performance are two key performance measures. However, we observe two notable issues in existing metrics: 1) manually delaying the delivery of some tokens can improve SLOs, and 2) actively abandoning requests that do not meet SLOs can improve SLMs, both of which are counterintuitive. In this paper, we revisit SLOs and SLMs in LLM serving, and propose a new SLO that aligns with user experience. Based on the SLO, we propose a comprehensive metric framework called smooth goodput, which integrates SLOs and SLMs to reflect the nature of user experience in LLM serving. Through this unified framework, we reassess the performance of different LLM serving systems under multiple workloads. Evaluation results show that our metric framework provides a more comprehensive view of token delivery and request processing, and effectively captures the optimal point of user experience and system performance with different serving strategies.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 39 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

12 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  2. 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines8 Aug
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 6

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  3. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  6. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  2. 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloadingcited by 2
  3. 2024Software Performance Engineering for Foundation Model-Powered Softwarecited by 0
  4. 2026TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inferencecited by 0
  5. 2026ExaServe: Large-Scale LLM Serving on Exascale HPC Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMs
  2. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
  3. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
  4. 2024Queue management for slo-oriented large language model serving
  5. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  6. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  7. 2026Human-Less LLM Serving: Quantifying the Human Tax on Throughput
  8. 2026A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing