mlsanityData to 14 Oct 2026

JITServe: SLO-aware LLM Serving with Imprecise Request Information

arXiv 2504.20068 ↗First posted 24 Apr 2025cs.DC cs.LG cs.SYcited by 17

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The integration of Large Language Models (LLMs) into applications ranging from interactive chatbots to multi-agent systems has introduced a wide spectrum of service-level objectives (SLOs) for responsiveness. These include latency-sensitive requests emphasizing per-token latency in streaming chat, deadline-sensitive requests requiring rapid full responses to trigger external tools, and compound requests with evolving dependencies across multiple LLM calls. Despite-or perhaps, because of-this workload diversity and unpredictable request information (e.g., response lengths and dependencies), existing request schedulers have focused on aggregate performance, unable to ensure application-level SLO needs. This paper presents JITServe, the first SLO-aware LLM serving system designed to maximize service goodput (e.g., the number of tokens meeting request SLOs) across diverse workloads. JITServe novelly schedules requests using imprecise request information and gradually relaxes this conservatism by refining request information estimates as generation progresses. It applies a grouped margin goodput maximization algorithm to allocate just enough serving bandwidth to satisfy each request's SLO just-in-time (JIT), maximizing residual capacity for others, while deciding the composition of requests in a batch to maximize efficiency and goodput with provable guarantees. Our evaluation across diverse realistic workloads, including chat, deep research, and agentic pipelines, shows that JITServe improves service goodput by 1.4x-6.3x, alternatively achieving 28.5%-83.2% resource savings, compared to state-of-the-art designs.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 78 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

43 of 78 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services25 Apr
  4. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023Fairness in Serving Large Language Models31 Dec
  8. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  9. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May
  10. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
21 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  8. 2019A Survey on Graph Kernelscited by 148
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  10. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 101
  11. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  12. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  13. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  14. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  15. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  16. 2024Understanding User Experience in Large Language Model Interactionscited by 44
  17. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  18. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  19. 2024LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDscited by 19
  20. 2024Building AI Agents for Autonomous Clouds: Challenges and Design Principlescited by 19
  21. 2016K-Medoids For K-Means Seedingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 14.

17 papers.

Sort:

By area:

  1. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  3. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 11
  4. 2025PolyServe: Efficient Multi-SLO Serving at Scalecited by 5
  5. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  6. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inferencecited by 1
  7. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  8. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  9. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  10. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 1
  11. 2026SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systemscited by 1
  12. 2026LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflowscited by 0
5 more
  1. 2026Lodestar: An Online-Learning LLM Inference Routercited by 0
  2. 2026FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Servingcited by 0
  3. 2026ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platformcited by 0
  4. 2026Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflowscited by 0
  5. 2026REVA: Reusable Evidence View Aggregation for Context-Efficient RAG Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
  2. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
  3. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  4. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMs
  5. 2026SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Services
  6. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
  7. 2024Queue management for slo-oriented large language model serving
  8. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving