Semantic Scheduling for LLM Inference
Not on the map. Why?
Abstract (arXiv)
Conventional operating system scheduling algorithms are largely content-ignorant, making decisions based on factors such as latency or fairness without considering the actual intents or semantics of processes. Consequently, these algorithms often do not prioritize tasks that require urgent attention or carry higher importance, such as in emergency management scenarios. However, recent advances in language models enable semantic analysis of processes, allowing for more intelligent and context-aware scheduling decisions. In this paper, we introduce the concept of semantic scheduling in scheduling of requests from large language models (LLM), where the semantics of the process guide the scheduling priorities. We present a novel scheduling algorithm with optimal time complexity, designed to minimize the overall waiting time in LLM-based prompt scheduling. To illustrate its effectiveness, we present a medical emergency management application, underscoring the potential benefits of semantic scheduling for critical, time-sensitive tasks. The code and data are available at https://github.com/Wenyueh/latency_optimization_with_priority_constraints.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 13 Jun 2025.
Builds on
26 of 41 references.
References · 26
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
- 2023Efficient Large Language Models: A Surveycited by 186
- 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
- 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
- 2023Online Speculative Decodingcited by 113
14 more
- 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
- 2024AIOS: LLM Agent Operating Systemcited by 97
- 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
- 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
- 2024Efficient LLM Scheduling by Learning to Rankcited by 85
- 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
- 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
- 2024LLM-Assisted Crisis Management: Building Advanced LLM Platforms for Effective Emergency Response and Public Collaborationcited by 21
- 2024On Large Language Models in National Security Applicationscited by 19
- 2024Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interfacecited by 16
- 2024Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical educationcited by 13
- 2024Fast Inference for Augmented Large Language Modelscited by 10
- 2023Optimizing National Security Strategies through LLM-Driven Artificial Intelligence Integrationcited by 6
- 2024OCCAM: Towards Cost-Efficient and Accuracy-Aware Classification Inferencecited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.