mlsanityData to 14 Oct 2026

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models

arXiv 2507.14204 ↗First posted 14 Jul 2025cs.LG cs.AI cs.CLcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have spurred interest in numerous applications requiring robust long-range capabilities, essential for processing extensive input contexts and continuously generating extended outputs. As sequence lengths increase, the number of Key-Value (KV) pairs in LLMs escalates, creating a significant efficiency bottleneck. In this paper, we propose a new KV cache optimization paradigm called LaCache, a training-free method for efficient and accurate generative inference of LLMs. LaCache enables LLMs to simultaneously address both of the critical challenges in long-range modeling: robust long-range capabilities and continuous generation without running out-of-memory (OOM). Specifically, LaCache integrates two key innovations: (1) a ladder-shaped KV cache pattern that stores KV pairs not only sequentially (left-to-right within each layer) but also across layers (from shallow to deep), providing an extended span for capturing long-range dependencies under a fixed storage budget, thereby boosting long-range capabilities; and (2) an iterative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within a fixed cache size. This token distance-based dynamic compression enables more effective continuous generation under constrained cache budgets. Experiments across various tasks, benchmarks, and LLM models consistently validate LaCache's effectiveness in enhancing LLMs' long-range capabilities. Our code is available at https://github.com/GATECH-EIC/LaCache.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 47 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

39 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024Effectively Compress KV Heads for LLM11 Jun
  4. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2023Mistral 7Bcited by 3,164
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
19 more
  1. 2020Reformer: The Efficient Transformercited by 1,668
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  11. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  12. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  13. 2024Transformers are Multi-State RNNscited by 120
  14. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
  15. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  16. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  17. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 42
  18. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  19. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 85
  2. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  3. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  4. 2026ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothingcited by 9
  5. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  6. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  7. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1
  8. 2026Adaptive Mass-Segmented KV Compression for Long-Context Reasoningcited by 1
  9. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  3. 2024Efficient LLM Inference with Kcache
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  5. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review