mlsanityData to 14 Oct 2026

Efficient Serving of LLM Applications with Probabilistic Demand Modeling

arXiv 2506.14851 ↗First posted 17 Jun 2025cs.DC cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Applications based on Large Language Models (LLMs) contains a series of tasks to address real-world problems with boosted capability, which have dynamic demand volumes on diverse backends. Existing serving systems treat the resource demands of LLM applications as a blackbox, compromising end-to-end efficiency due to improper queuing order and backend warm up latency. We find that the resource demands of LLM applications can be modeled in a general and accurate manner with Probabilistic Demand Graph (PDGraph). We then propose Hermes, which leverages PDGraph for efficient serving of LLM applications. Confronting probabilistic demand description, Hermes applies the Gittins policy to determine the scheduling order that can minimize the average application completion time. It also uses the PDGraph model to help prewarm cold backends at proper moments. Experiments with diverse LLM applications confirm that Hermes can effectively improve the application serving efficiency, reducing the average completion time by over 70% and the P95 completion time by over 80%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 84 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

44 of 84 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  9. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
32 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  4. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  5. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  6. 2023Large Language Models in Finance: A Surveycited by 226
  7. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  8. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  10. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  11. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  13. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  14. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  15. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  16. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  17. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  18. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  19. 2023Fairness in Serving Large Language Modelscited by 78
  20. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 73
  21. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  22. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  23. 2024LLM With Tools: A Surveycited by 61
  24. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  25. 2024Queue management for slo-oriented large language model servingcited by 36
  26. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35
  27. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 33
  28. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 22
  29. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  30. 2022Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learningcited by 12
  31. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  32. 2024Ensuring Fair LLM Serving Amid Diverse Applicationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0