mlsanityData to 14 Oct 2026

Jenga: Effective Memory Management for Serving LLM with Heterogeneity

arXiv 2503.18292 ↗First posted 24 Mar 2025cs.DCcited by 25

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory efficiently is crucial. While PagedAttention has recently been proposed to improve the efficiency of memory management, we find that the growing heterogeneity in the embeddings dimensions, attention, and access patterns of modern LLM architectures introduces new challenges for memory allocation. In this paper, we present Jenga, a novel memory allocation framework for heterogeneous embeddings in LLMs. Jenga tackles two key challenges: (1) minimizing memory fragmentation when managing embeddings of different sizes, and (2) enabling flexible caching and eviction policies tailored to the specific token-dependency patterns of various layers. Jenga employs a two-level memory allocator, leveraging the least common multiple (LCM) of embedding sizes to optimize memory usage and providing APIs to express layer-specific caching logic to enhance memory reuse. We implemente Jenga on vLLM, a state-of-the-art LLM inference engine, and evaluate it with diverse LLMs, datasets, and GPU configurations. Evaluations show that Jenga improves GPU memory utilization by up to 79.6%, and increases serving throughput by up to 4.92x (1.80x on average).

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 63 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

42 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  2. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  4. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
25 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  13. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  14. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  15. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  16. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  17. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  18. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  19. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  20. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  21. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  22. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  23. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  24. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  25. 2024GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitchingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 21.

25 papers.

Sort:

By area:

  1. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  2. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 10
  3. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  4. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  5. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  6. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  7. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  8. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
  9. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  10. 2026Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Cachescited by 3
  11. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  12. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
13 more
  1. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharingcited by 2
  2. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  3. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1
  4. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Servingcited by 1
  5. 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inferencecited by 0
  6. 2026SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Servingcited by 0
  7. 2026Characterizing Software Aging in GPU-Based LLM Serving Systemscited by 0
  8. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  9. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructurecited by 0
  10. 2026BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inferencecited by 0
  11. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Cachingcited by 0
  12. 2026WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reusecited by 0
  13. 2026Scaling LLM Inference via Parallelism: Throughput-Latency Trade-offcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  4. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
  5. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  6. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  7. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  8. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference