# Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

arXiv: https://arxiv.org/abs/2503.08311 · HTML: https://mlsanity.com/paper/2503.08311/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [S3: Increasing GPU Utilization during Generative Inference for Higher Throughput](https://mlsanity.com/paper/2306.06000/) (2023)
- [LLM Inference Serving: Survey of Recent Advances and Opportunities](https://mlsanity.com/paper/2407.12391/) (2024)
- [Towards Pareto Optimal Throughput in Small Language Model Serving](https://mlsanity.com/paper/2404.03353/) (2024)

## Built on by (most cited first, 30 of 33)

- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [BroRL: Scaling Reinforcement Learning via Broadened Exploration](https://mlsanity.com/paper/2510.01180/) (2025)
- [ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models](https://mlsanity.com/paper/2510.01290/) (2025)
- [SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding](https://mlsanity.com/paper/2603.18567/) (2026)
- [KVzap: Fast, Adaptive, and Faithful KV Cache Pruning](https://mlsanity.com/paper/2601.07891/) (2026)
- [Hardware-Efficient Attention for Fast Decoding](https://mlsanity.com/paper/2505.21487/) (2025)
- [Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures](https://mlsanity.com/paper/2604.09048/) (2026)
- [Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective](https://mlsanity.com/paper/2604.25975/) (2026)
- [Compute-Optimal Quantization-Aware Training](https://mlsanity.com/paper/2509.22935/) (2025)
- [Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective](https://mlsanity.com/paper/2511.00739/) (2025)
- [Adaptive Block-Scaled Data Types](https://mlsanity.com/paper/2603.28765/) (2026)
- [Toward Robust and Efficient ML-Based GPU Caching for Modern Inference](https://mlsanity.com/paper/2509.20979/) (2025)
- [Equinox: Holistic Fair Scheduling in Serving Large Language Models](https://mlsanity.com/paper/2508.16646/) (2025)
- [A Data-driven ML Approach for Maximizing Performance in LLM-Adapter Serving](https://mlsanity.com/paper/2508.08343/) (2025)
- [Mesh RAG: Retrieval Augmentation for Autoregressive Mesh Generation](https://mlsanity.com/paper/2511.16807/) (2025)
- [QMC: Efficient SLM Edge Inference via Outlier-Aware Quantization and Emergent Memories Co-Design](https://mlsanity.com/paper/2601.14549/) (2026)
- [MARLIN: Multi-Agent Game-Theoretic Reinforcement Learning for Sustainable LLM Inference in Cloud Datacenters](https://mlsanity.com/paper/2605.13496/) (2026)
- [Intra-Prompt Parallel Decoding for Common-Context Question Answering](https://mlsanity.com/paper/2609.05707/) (2026)
- [Direct Multi-Token Decoding](https://mlsanity.com/paper/2510.11958/) (2025)
- [PICNIC: Silicon Photonic Interconnected Chiplets with Computational Network and In-memory Computing for LLM Inference Acceleration](https://mlsanity.com/paper/2511.04036/) (2025)
- [Personalized Medication Planning via Direct Domain Modeling and LLM-Generated Heuristics](https://mlsanity.com/paper/2601.03687/) (2026)
- [Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving](https://mlsanity.com/paper/2602.24044/) (2026)
- [SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity](https://mlsanity.com/paper/2603.07917/) (2026)
- [Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study](https://mlsanity.com/paper/2603.10031/) (2026)
- [Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving](https://mlsanity.com/paper/2604.15732/) (2026)
- [Threshold-Based Exclusive Batching for LLM Inference](https://mlsanity.com/paper/2606.00516/) (2026)
- [Lights, Camera, Carbon: Architectural Scaling Laws for Video Generation Energy Consumption](https://mlsanity.com/paper/2607.04553/) (2026)
- [Think Before You Grid-Search: Floor-First Triage for LLM Serving](https://mlsanity.com/paper/2607.05876/) (2026)
- [The Economics of AI Decoding Chips: Rebalancing Compute, Capacity, and Bandwidth for Efficient LLM Inference](https://mlsanity.com/paper/2607.13068/) (2026)
- [NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement](https://mlsanity.com/paper/2607.26633/) (2026)

## Abstract (arXiv)

Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference. While batching is commonly used to increase throughput, performance gains plateau beyond a certain batch size, especially with smaller models, a phenomenon that existing literature typically explains as a shift to the compute-bound regime. In this paper, through an in-depth GPU-level analysis, we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck. To address this, we propose a Batching Configuration Advisor (BCA) that optimizes memory allocation, reducing GPU memory requirements with minimal impact on throughput. The freed memory and underutilized GPU compute capabilities can then be leveraged by concurrent workloads. Specifically, we use model replication to improve serving throughput and GPU utilization. Our findings challenge conventional assumptions about LLM inference, offering new insights and practical strategies for improving resource utilization, particularly for smaller language models. The code is publicly available at https://github.com/FerranAgulloLopez/vLLMBatchingMemoryGap.