mlsanityData to 14 Oct 2026

Preble: Efficient Distributed Prompt Scheduling for LLM Serving

arXiv 2407.00023 ↗First posted 8 May 2024cs.DC cs.LGcited by 70

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Prompts to large language models (LLMs) have evolved beyond simple user questions. For LLMs to solve complex problems, today's practices are to include domain-specific instructions, illustration of tool usages, and/or long context such as textbook chapters in prompts. As such, many parts of prompts are repetitive across requests. Recent works propose to cache and reuse KV state of prompts. However, they are all confined to a single-GPU optimization, while production LLM serving systems are distributed by nature. This paper proposes Preble, the first distributed LLM serving platform that targets and optimizes for prompt sharing. We designed a distributed scheduling system that co-optimizes KV state reuse and computation load-balancing with a new scheduling algorithm and a hierarchical scheduling mechanism. Our evaluation of Preble with real workloads and request arrival patterns on two open-source LLMs shows that Preble outperforms the SOTA serving systems by 1.5X to 14.5X on average latency and 2X to 10X on p99 latency.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 69 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

36 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  4. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
20 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 2021Measuring Coding Challenge Competence With APPScited by 902
  3. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  9. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  12. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  13. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  14. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  15. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  16. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  17. 2024More Agents Is All You Needcited by 180
  18. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  19. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  20. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 19; 2026: 47.

70 papers.

Sort:

By area:

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  2. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  3. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  4. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
  5. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  6. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  7. 2024Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interfacecited by 16
  8. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  9. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  10. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16
  11. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 14
  12. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 13
38 more
  1. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scalecited by 13
  2. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  3. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  4. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  5. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  6. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  7. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  8. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  9. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  10. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Servingcited by 4
  11. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  12. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  13. 2026Diffusion Templates: A Unified Plugin Framework for Controllable Diffusioncited by 3
  14. 2026Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Cachescited by 3
  15. 2026ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Servingcited by 3
  16. 2025Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inferencecited by 2
  17. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  18. 2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 2
  19. 2026Coordinated Scheduling for MoE LLM Servingcited by 2
  20. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  21. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  22. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 2
  23. 2025PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Servingcited by 1
  24. 2026Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsitycited by 1
  25. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1
  26. 2026GORGO: Online Tuning for Cross-Region Network-Aware LLM Servingcited by 1
  27. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoningcited by 1
  28. 2026DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72cited by 1
  29. 2026Enabling KV Caching of Shared Prefix for Diffusion Language Modelscited by 1
  30. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 1
  31. 2026Scheduling Mixed RL Rollouts Beyond Prefix Localitycited by 1
  32. 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Servingcited by 1
  33. 2026Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerfcited by 1
  34. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Servingcited by 1
  35. 2026KV2: A Self-Refining KV Cachecited by 1
  36. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  37. 2025OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latencycited by 0
  38. 2026RadixMLP -- Intra-batch Deduplication for Causal Transformerscited by 0

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  2. 2025Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
  3. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Serving
  4. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  5. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees
  6. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  8. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice