mlsanityData to 7 Oct 2026

Serving systems and request scheduling

402 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: llm, serving, llm serving, inference, latency

Key facts. Serving systems and request scheduling: 402 papers on arXiv since 2021 (data through 7 Oct 2026). 178 of its 402 mapped papers (44%) cite “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving” (2024, arXiv 2401.09670). 153 of its 402 mapped papers (38%) cite “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve” (2024, arXiv 2403.02310). 12 new papers 24 Sep–7 Oct.

What’s new

24 Sep – 7 Oct 2026

12 new papers, 24 Sep – 7 Oct; 0 this week.

  1. Earlier · 12
  2. Beyond LLM Serving: Characterizing Vision-Language-Action Workloads for Embodied AI System Design also fits: Efficient reasoning and agent inference
  3. FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
  4. Characterizing High Bandwidth Flash for LLM Serving
  5. Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure also fits: Efficient reasoning and agent inference
  6. SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving also fits: KV cache optimization
  7. vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains
  8. Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  9. DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
  10. Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving
  11. EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models also fits: Early exit and edge inference
All 12
  1. DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving also fits: Efficient reasoning and agent inference
  2. Cross-Model Autoscaling for Shared LLM Serving

Foundations

Most cited by papers here

  1. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving2024cited by 178 here
  2. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve2024cited by 153 here
  3. Llumnix: Dynamic Scheduling for Large Language Model Serving2024cited by 72 here
  4. Fast Distributed Inference Serving for Large Language Models2023cited by 63 here
  5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving2024cited by 61 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 65 here
  2. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 25 here
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 23 here
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 39 here
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 72 here
19 more it builds on, not in this chain
  1. 2023Fast Distributed Inference Serving for Large Language Modelscited by 63 here
  2. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 31 here
  3. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 33 here
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 24 here
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 51 here
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 289 here
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 30 here
  8. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 45 here
  9. 2023Fairness in Serving Large Language Modelscited by 31 here
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 178 here
  11. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 22 here
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 153 here
  13. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 32 here
  14. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 23 here
  15. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 32 here
  16. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 41 here
  17. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 61 here
  18. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 28 here
  19. 2024Efficient LLM Scheduling by Learning to Rankcited by 32 here

The 24 most cited of 450.

Browse

30 of 402, ranked 6–35

  1. 6SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills202351
  2. 7SpotServe: Serving Generative Large Language Models on Preemptible Instances202345
  3. 8LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism202441
  4. 9BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems202439
  5. 10Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction202432
  6. 11Efficient LLM Scheduling by Learning to Rank202432
  7. 12Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline202331
  8. 13Fairness in Serving Large Language Models202331
  9. 14MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving202423
  10. 15vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention202420
All 30
  1. 16NanoFlow: Towards Optimal Large Language Model Serving Throughput202420
  2. 17FlashDecoding++: Faster Large Language Model Inference on GPUs202320
  3. 18Queue management for slo-oriented large language model serving202419
  4. 19MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool202419
  5. 20Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services202418
  6. 21Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache202417
  7. 22LLM Inference Unveiled: Survey and Roofline Model Insights202417
  8. 23Preble: Efficient Distributed Prompt Scheduling for LLM Serving202416
  9. 24A Survey on Efficient Inference for Large Language Models202415
  10. 25P/D-Serve: Serving Disaggregated Large Language Model at Scale202415
  11. 26LLM in a flash: Efficient Large Language Model Inference with Limited Memory202314
  12. 27ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments202513
  13. 28Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems202313
  14. 29Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs202512
  15. 30ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production202512
  16. 31InferCept: Efficient Intercept Support for Augmented Large Language Model Inference202410
  17. 32FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines202410
  18. 33SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving20249
  19. 34SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling20258
  20. 35semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage20258