mlsanityData to 14 Oct 2026

AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications

arXiv 2503.13737 ↗First posted 17 Mar 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider a mixed-prompt scenario for a large language model (LLM) inference serving system that supports diverse applications with both short prompts and long prompts and heterogeneous SLOs for iteration time. To improve throughput when handling long prompts, previous research introduces a chunking method, but has not addressed heterogeneous SLOs. To address the limitation, we propose AccelGen, a high-throughput LLM inference serving system with heterogeneous SLO guarantees for diverse applications. AccelGen introduces four core components: (1) SLO-guaranteed dynamic chunking, which dynamically adjusts chunk sizes to maximize GPU compute utilization while meeting iteration-level SLOs; (2) Iteration-level SLO-based task prioritization, which prioritizes tight-SLO requests and batches requests with similar SLOs; (3) Multi-resource-aware batching, which selects queued requests to maximize the utilizations of both GPU compute resource and key-value cache (KVC). Trace-driven real experiments demonstrate that AccelGen achieves 1.42-11.21X higher throughput, 1.43-13.71X higher goodput, 37-90% higher SLO attainment, and 1.61-12.22X lower response latency compared to the state-of-the-art approaches. It achieves performance near the Oracle, which optimally maximizes goodput.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

24 of 40 references.

References · 24

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2020Recipes for building an open-domain chatbotcited by 593
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
12 more
  1. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  3. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  4. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  5. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  6. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  7. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  8. 2023Fairness in Serving Large Language Modelscited by 78
  9. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 52
  10. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  11. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  12. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  2. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  2. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  3. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  4. 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  6. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  7. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  8. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving