mlsanityData to 14 Oct 2026

Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

arXiv 2503.08311 ↗First posted 11 Mar 2025cs.DC cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference. While batching is commonly used to increase throughput, performance gains plateau beyond a certain batch size, especially with smaller models, a phenomenon that existing literature typically explains as a shift to the compute-bound regime. In this paper, through an in-depth GPU-level analysis, we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck. To address this, we propose a Batching Configuration Advisor (BCA) that optimizes memory allocation, reducing GPU memory requirements with minimal impact on throughput. The freed memory and underutilized GPU compute capabilities can then be leveraged by concurrent workloads. Specifically, we use model replication to improve serving throughput and GPU utilization. Our findings challenge conventional assumptions about LLM inference, offering new insights and practical strategies for improving resource utilization, particularly for smaller language models. The code is publicly available at https://github.com/FerranAgulloLopez/vLLMBatchingMemoryGap.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

24 of 32 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
12 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  9. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  10. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  11. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  12. 2024Towards Pareto Optimal Throughput in Small Language Model Servingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 21.

33 papers.

Sort:

By area:

  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  2. 2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 25
  3. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  4. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 16
  5. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 15
  6. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  7. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architecturescited by 5
  8. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 5
  9. 2025Compute-Optimal Quantization-Aware Trainingcited by 4
  10. 2025Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspectivecited by 4
  11. 2026Adaptive Block-Scaled Data Typescited by 4
  12. 2025Toward Robust and Efficient ML-Based GPU Caching for Modern Inferencecited by 3
21 more
  1. 2025Equinox: Holistic Fair Scheduling in Serving Large Language Modelscited by 2
  2. 2025A Data-driven ML Approach for Maximizing Performance in LLM-Adapter Servingcited by 1
  3. 2025Mesh RAG: Retrieval Augmentation for Autoregressive Mesh Generationcited by 1
  4. 2026QMC: Efficient SLM Edge Inference via Outlier-Aware Quantization and Emergent Memories Co-Designcited by 1
  5. 2026MARLIN: Multi-Agent Game-Theoretic Reinforcement Learning for Sustainable LLM Inference in Cloud Datacenterscited by 1
  6. 2026Intra-Prompt Parallel Decoding for Common-Context Question Answeringcited by 1
  7. 2025Direct Multi-Token Decodingcited by 0
  8. 2025PICNIC: Silicon Photonic Interconnected Chiplets with Computational Network and In-memory Computing for LLM Inference Accelerationcited by 0
  9. 2026Personalized Medication Planning via Direct Domain Modeling and LLM-Generated Heuristicscited by 0
  10. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Servingcited by 0
  11. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybriditycited by 0
  12. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Studycited by 0
  13. 2026Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Servingcited by 0
  14. 2026Threshold-Based Exclusive Batching for LLM Inferencecited by 0
  15. 2026Lights, Camera, Carbon: Architectural Scaling Laws for Video Generation Energy Consumptioncited by 0
  16. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  17. 2026The Economics of AI Decoding Chips: Rebalancing Compute, Capacity, and Bandwidth for Efficient LLM Inferencecited by 0
  18. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placementcited by 0
  19. 2026SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Servingcited by 0
  20. 2026Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservationcited by 0
  21. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0