mlsanityData to 7 Oct 2026

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

arXiv 2308.16369 ↗First posted 31 Aug 2023cs.LG cs.DCcited by 187

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Model (LLM) inference consists of two distinct phases - prefill phase which processes the input prompt and decode phase which generates output tokens autoregressively. While the prefill phase effectively saturates GPU compute at small batch sizes, the decode phase results in low compute utilization as it generates one token at a time per request. The varying prefill and decode times also lead to imbalance across micro-batches when using pipeline parallelism, resulting in further inefficiency due to bubbles. We present SARATHI to address these challenges. SARATHI employs chunked-prefills, which splits a prefill request into equal sized chunks, and decode-maximal batching, which constructs a batch using a single prefill chunk and populates the remaining slots with decodes. During inference, the prefill chunk saturates GPU compute, while the decode requests 'piggyback' and cost up to an order of magnitude less compared to a decode-only batch. Chunked-prefills allows constructing multiple decode-maximal batches from a single prefill request, maximizing coverage of decodes that can piggyback. Furthermore, the uniform compute design of these batches ameliorates the imbalance between micro-batches, significantly reducing pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware. For the LLaMA-13B model on A6000 GPU, SARATHI improves decode throughput by up to 10x, and accelerates end-to-end throughput by up to 1.33x. For LLaMa-33B on A100 GPU, we achieve 1.25x higher end-to-end-throughput and up to 4.25x higher decode throughput. When used with pipeline parallelism on GPT-3, SARATHI reduces bubbles by 6.29x, resulting in an end-to-end throughput improvement of 1.91x.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 39 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

23 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
10 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  7. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  8. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  9. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  10. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 49; 2025: 58; 2026: 77.

187 papers.

Sort:

By area:

  1. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  5. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  6. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  7. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  9. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  10. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  12. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
38 more
  1. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  2. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  3. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  4. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 57
  5. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  6. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  7. 2025LongCat-Flash Technical Reportcited by 54
  8. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  9. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  10. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  11. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  12. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  13. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  14. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 32
  15. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  16. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
  17. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  18. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  19. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  20. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  21. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  22. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  23. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  24. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  25. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  26. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 17
  27. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  28. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16
  29. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  30. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  31. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  32. 2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Executioncited by 14
  33. 2024Accelerating Retrieval-Augmented Generationcited by 14
  34. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  35. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  36. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  37. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  38. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  3. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
  4. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  5. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  6. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  7. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  8. 2025HAMburger: Accelerating LLM Inference via Token Smashing