mlsanityData to 7 Oct 2026

Llumnix: Dynamic Scheduling for Large Language Model Serving

arXiv 2406.03243 ↗First posted 5 Jun 2024cs.AR cs.DC cs.LGcited by 167

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Inference serving for large language models (LLMs) is the key to unleashing their potential in people's daily lives. However, efficient LLM serving remains challenging today because the requests are inherently heterogeneous and unpredictable in terms of resource and latency requirements, as a result of the diverse applications and the dynamic execution nature of LLMs. Existing systems are fundamentally limited in handling these characteristics and cause problems such as severe queuing delays, poor tail latencies, and SLO violations. We introduce Llumnix, an LLM serving system that reacts to such heterogeneous and unpredictable requests by runtime rescheduling across multiple model instances. Similar to context switching across CPU cores in modern operating systems, Llumnix reschedules requests to improve load balancing and isolation, mitigate resource fragmentation, and differentiate request priorities and SLOs. Llumnix implements the rescheduling with an efficient and scalable live migration mechanism for requests and their in-memory states, and exploits it in a dynamic scheduling policy that unifies the multiple rescheduling scenarios elegantly. Our evaluations show that Llumnix improves tail latencies by an order of magnitude, accelerates high-priority requests by up to 1.5x, and delivers up to 36% cost savings while achieving similar tail latencies, compared against state-of-the-art LLM serving systems. Llumnix is publicly available at https://github.com/AlibabaPAI/llumnix.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 46 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

20 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  3. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
5 more
  1. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  2. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  3. 2022Can Foundation Models Wrangle Your Data?cited by 173
  4. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  5. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 71; 2026: 86.

167 papers.

Sort:

By area:

  1. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  2. 2025Astra: A Multi-Agent System for GPU Kernel Performance Optimizationcited by 46
  3. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  4. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  5. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  6. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  7. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  8. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  9. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  10. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  11. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 17
  12. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
38 more
  1. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  2. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
  3. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Cloudscited by 12
  4. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  5. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  6. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  7. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 10
  8. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  9. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  10. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  11. 2025SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Modelscited by 9
  12. 2025FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusterscited by 9
  13. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 9
  14. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 8
  15. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  16. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  17. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
  18. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 7
  19. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  20. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  21. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  22. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Servingcited by 5
  23. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 5
  24. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  25. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  26. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5
  27. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 5
  28. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 5
  29. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  30. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
  31. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  32. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Servingcited by 4
  33. 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMscited by 4
  34. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 4
  35. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decodingcited by 4
  36. 2024PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inferencecited by 3
  37. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Cachingcited by 3
  38. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchronycited by 3

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  2. 2025Niyama : Breaking the Silos of LLM Inference Serving
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  4. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  5. 2024Fast Inference for Augmented Large Language Models
  6. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  7. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  8. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference