mlsanityData to 7 Oct 2026

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

arXiv 2403.02310 ↗First posted 4 Mar 2024cs.LG cs.DCcited by 507

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per request. This makes batching highly effective for decodes and consequently for overall throughput. However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency. We introduce an efficient LLM inference scheduler, Sarathi-Serve, to address this throughput-latency tradeoff. Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency. Furthermore, uniform batches in Sarathi-Serve ameliorate the imbalance between iterations resulting in minimal pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware under tail latency constraints. For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity and up to 3.7x higher serving capacity for the Yi-34B model on two A100 GPUs as compared to vLLM. When used with pipeline parallelism on Falcon-180B, Sarathi-Serve provides up to 5.6x gain in the end-to-end serving capacity. The source code for Sarathi-Serve is available at https://github.com/microsoft/sarathi-serve.

Key facts. Cited by 507 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 62 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

30 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fairness in Serving Large Language Models31 Dec
  2. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023Mistral 7Bcited by 3,065
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
16 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  8. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  9. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  10. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  11. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  12. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  13. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  14. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  15. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  16. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 71; 2025: 163; 2026: 273.

507 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  4. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  5. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  6. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  7. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 97
  8. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  9. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  10. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  11. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  12. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
38 more
  1. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  2. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  3. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  4. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  5. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  6. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  7. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  8. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  9. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  10. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  11. 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspectivecited by 38
  12. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  13. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  14. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  15. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  16. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  17. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  18. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  19. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  20. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 31
  21. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  22. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  23. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  24. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  25. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  26. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  27. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  28. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  29. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  30. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  31. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  32. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  33. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  34. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  35. 2024LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferencescited by 19
  36. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  37. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  38. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18

50 most cited of 507 All 507 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  2. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  3. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
  4. 2024A System for Microserving of LLMs
  5. 2026SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
  6. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  7. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
  8. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving