mlsanityData to 13 Oct 2026

EconoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Serving

arXiv 2411.06364 ↗First posted 10 Nov 2024cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As Large Language Models (LLMs) continue to grow, reducing costs and alleviating GPU demands has become increasingly critical. However, existing schedulers primarily target either GPU compute or Key-Value Cache (KVC) utilization, failing to fully optimize both GPU compute and KVC usage during each iteration or guarantee timely KVC allocations when needed. To address these challenges, we conducted a trace-based experimental analysis and made insightful observations, leading to the design of a system called EconoServe. EconoServe maximizes multi-resource utilization while ensuring service-level objective (SLO) guarantees in LLM serving. To enable adding prompts to a batch to maximize GPU utilization in each iteration, EconoServe maintains separate waiting queues for prompt processing tasks (PTs) and generation tasks (GTs). It batches GTs with the same predicted response lengths (RL) to save scheduling time and allocates KVC space for the predicted RL to avoid KVC allocation failures. It further has a novel KVC pipelining method, allowing sharing allocated but unused KVC space to enhance KVC utilization. In addition, it prioritizes queued requests that occupy more KVC to release KVC earlier and satisfy request service-level-objective (SLO). Experimental results demonstrate that EconoServe increases throughput by up to 4× with the same level of latency, generates up to 91% lower job completion time and up to 91% higher SLO satisfaction ratio compared to vLLM. It also reduces the number of GPUs used in DistServe by up to 78% while maintaining the same level of goodput.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 41 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

23 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023Fairness in Serving Large Language Models31 Dec
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  5. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 545
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 432
  12. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 250
6 more
  1. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  2. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 116
  3. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  4. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  5. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  6. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  3. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  4. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
  5. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
  6. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
  7. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  8. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters