mlsanityData to 7 Oct 2026

BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching

arXiv 2412.03594 ↗First posted 29 Nov 2024cs.CL cs.AI cs.DCcited by 31

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) increasingly play an important role in a wide range of information processing and management tasks in industry. Many of these tasks are performed in large batches or even offline, and the performance indicator for which is throughput. These tasks usually show the characteristic of prefix sharing, where different prompt input can partially show the common prefix. However, the existing LLM inference engines tend to optimize the streaming requests and show limitations of supporting the large batched tasks with the prefix sharing characteristic. The existing solutions use the LRU-based cache to reuse the KV context of common prefix between requests. The KV context that are about to be reused may be prematurely evicted with the implicit cache management. Besides, the streaming oriented systems do not leverage the request-batch information and can not mix the decoding tokens with the prefill chunks to the best for the batched scenarios, and thus fails to saturate the GPU. We propose BatchLLM to address the above problems. BatchLLM explicitly identifies the common prefixes globally. The requests sharing the same prefix will be scheduled together to reuse the KV context the best. BatchLLM reorders the requests and schedules the requests with larger ratio of decoding first to better mix the decoding tokens with the latter prefill chunks, and applies memory-centric token batching to enlarge the token-batch sizes, which helps to increase the GPU utilization. Extensive evaluation shows that BatchLLM outperforms vLLM and SGLang by 1.3× to 10.8× on a set of microbenchmarks and a typical industry workload under different hardware environments. Code is available at https://github.com/microsoft/MixLLM/tree/batchllmvllm064.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

26 of 39 references.

References · 26

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  8. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  11. 2018Online normalizer calculation for softmaxcited by 154
  12. 2021Self-attention Does Not Need O(n2) Memorycited by 152
14 more
  1. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  2. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  3. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  4. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  5. 2023Fairness in Serving Large Language Modelscited by 77
  6. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  7. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  8. 2024Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender Systemcited by 50
  9. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  10. 2023Large Search Model: Redefining Search Stack in the Era of LLMscited by 18
  11. 2020Automatic Horizontal Fusion for GPU Kernelscited by 17
  12. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  13. 2024RelayAttention for Efficient Large Language Model Serving with Long System Promptscited by 13
  14. 2023Lingua Manga: A Generic Large Language Model Centric System for Data Curationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 17.

31 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inferencecited by 18
  3. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  4. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  5. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  6. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 10
  7. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 9
  8. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  9. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  10. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 5
  11. 2025Hybrid OCR-LLM Framework for Enterprise-Scale Document Information Extraction Under Copy-heavy Taskcited by 5
  12. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
19 more
  1. 2025A Survey of LLM × DATAcited by 4
  2. 2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systemscited by 4
  3. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  4. 2025FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoningcited by 3
  5. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
  6. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inferencecited by 1
  7. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1
  8. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 1
  9. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  10. 2026LaCache: Robust Semantic Caching for LLM Servingcited by 1
  11. 2026RadixMLP -- Intra-batch Deduplication for Causal Transformerscited by 0
  12. 2026LLM-CoOpt: A Co-Design and Optimization Framework for Efficient LLM Inference on Heterogeneous Platformscited by 0
  13. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  14. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Serverscited by 0
  15. 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inferencecited by 0
  16. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  17. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Servingcited by 0
  18. 2026PromptPack: Scaling LLM Annotation Agents for Online Recommendationcited by 0
  19. 2026Inference Auctionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  2. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  3. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  4. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  5. 2026Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
  6. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  7. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  8. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference