mlsanityData to 7 Oct 2026

Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

arXiv 2403.09636 ↗First posted 14 Mar 2024cs.CLcited by 115

Not on the map. Why?

Cited by 70 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value representations for past tokens, whose size scales linearly with the input sequence length and batch size. As a solution, we propose Dynamic Memory Compression (DMC), a method for online key-value cache compression at inference time. Most importantly, the model learns to apply different compression ratios in different heads and layers. We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers, achieving up to 7x throughput increase during auto-regressive inference on an NVIDIA H100 GPU. DMC is applied via continued pre-training on a negligible percentage of the original data without adding any extra parameters. DMC preserves the original downstream performance with up to 4x cache compression, outperforming up-trained grouped-query attention (GQA) and key-value eviction policies (H2O, TOVA). GQA and DMC can be even combined to obtain compounded gains. Hence, DMC can serve as a drop-in replacement for KV caching in existing LLMs to fit longer contexts and larger batches within any given memory budget.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 40 of its 43 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

40 of 43 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
28 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2022Token Merging: Your ViT But Fastercited by 995
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  11. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  14. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  15. 2022Efficiently Scaling Transformer Inferencecited by 529
  16. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  17. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  18. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  19. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  20. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  21. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  22. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  23. 2024Transformers are Multi-State RNNscited by 120
  24. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  25. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  26. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  27. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  28. 2018Accelerating Neural Transformer via an Average Attention Networkcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 47; 2026: 36.

115 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  6. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  7. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  8. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  9. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  10. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  11. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  12. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
38 more
  1. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  2. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  3. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  4. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  5. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  6. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 44
  7. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  8. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  9. 2024A Survey of Mambacited by 39
  10. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  11. 2025PENCIL: Long Thoughts with Short Memorycited by 31
  12. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  13. 2025Cognitive Memory in Large Language Modelscited by 28
  14. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  15. 2025Inference-Time Hyper-Scaling with KV Cache Compressioncited by 28
  16. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  17. 2024A deeper look at depth pruning of LLMscited by 23
  18. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  19. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  20. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  21. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  22. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  23. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
  24. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 12
  25. 2026Learning to Evict from Key-Value Cachecited by 12
  26. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  27. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  28. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  29. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  30. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  31. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  32. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  33. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  34. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8
  35. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  36. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inferencecited by 8
  37. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inferencecited by 7
  38. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decodercited by 7

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  3. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  4. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  5. 2026One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2024LoMA: Lossless Compressed Memory Attention