mlsanityData to 14 Oct 2026

Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving

arXiv 2504.07494 ↗First posted 10 Apr 2025cs.LGcited by 6

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) inference serving systems are essential to various LLM-based applications. As demand for LLM services continues to grow, scaling these systems to handle high request rates while meeting latency Service-Level Objectives (SLOs), referred to as effective throughput, becomes critical. However, existing systems often struggle to improve effective throughput, primarily due to a significant decline in Time To First Token (TTFT) SLO attainment. We identify two major causes of this bottleneck: (1) memory-intensive KV cache that limits batch size expansion under GPU memory constraints, and (2) rigid batch composition enforced by the default First-Come-First-Serve scheduling policy. In this paper, we introduce Apt-Serve, a scalable framework designed to enhance effective throughput in LLM inference serving. Apt-Serve features a new hybrid cache scheme that combines KV cache with a memory-efficient hidden cache for reusable input hidden state vectors, allowing large batch sizes and improving request concurrency. Based on the hybrid cache, Apt-Serve employs an adaptive runtime scheduling mechanism that dynamically optimizes batch composition. We formally define the adaptive scheduling optimization problem and propose an efficient algorithm with theoretical guarantees. Extensive evaluations on three real-world datasets and LLMs ranging from 13B to 66B parameters demonstrate that Apt-Serve achieves up to 8.8x improvement in effective throughput compared to the state-of-the-art inference serving systems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 80 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

42 of 80 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  6. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
24 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  2. 2022Efficiently Scaling Transformer Inferencecited by 539
  3. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  4. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  6. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  7. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  8. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  9. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  10. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  11. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  12. 2021Accelerating Large Scale Real-Time GNN Inference using Channel Pruningcited by 41
  13. 2020Analyzing and Mitigating Data Stalls in DNN Trainingcited by 36
  14. 2019Database Meets Deep Learning: Challenges and Opportunitiescited by 30
  15. 2021HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Frameworkcited by 28
  16. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  17. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 14
  18. 2022Optimizing Machine Learning Inference Queries with Correlative Proxy Modelscited by 13
  19. 2020Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analyticscited by 11
  20. 2022Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelinescited by 10
  21. 2023Accelerating Scalable Graph Neural Network Inference with Node-Adaptive Propagationcited by 8
  22. 2024MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Trainingcited by 5
  23. 2023InferTurbo: A Scalable System for Boosting Full-graph Inference of Graph Neural Network over Huge Graphscited by 4
  24. 2024Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelinescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  2. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
  3. 2025HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scalingcited by 3
  4. 2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3
  5. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 3
  6. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving Systemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  2. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  3. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture
  4. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  5. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  6. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications
  7. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
  8. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments