mlsanityData to 7 Oct 2026

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

arXiv 2402.09398 ↗First posted 14 Feb 2024cs.LG cs.AIcited by 75

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Many computational factors limit broader deployment of large language models. In this paper, we focus on a memory bottleneck imposed by the key-value (KV) cache, a computational shortcut that requires storing previous KV pairs during decoding. While existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs to dramatically reduce the memory footprint of the cache, they can have limited success in tasks that require recollecting a majority of previous tokens. To alleviate this issue, we propose LESS, a simple integration of a (nearly free) constant sized cache with eviction-based cache methods, such that all tokens can be queried at later decoding steps. Its ability to retain information throughout time shows merit on a variety of tasks where we demonstrate LESS can help reduce the performance gap from caching everything, sometimes even matching it, all while being efficient. Relevant code can be found at https://github.com/hdong920/LESS.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 60 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

44 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2020Rethinking Attention with Performerscited by 1,584
30 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2015Teaching Machines to Read and Comprehendcited by 1,039
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2022Token Merging: Your ViT But Fastercited by 995
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2018Image Transformercited by 770
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  13. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  14. 2022Efficiently Scaling Transformer Inferencecited by 529
  15. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  16. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  17. 2021A Survey of Transformerscited by 399
  18. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  19. 2021Random Feature Attentioncited by 350
  20. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  21. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  22. 2024Transformers are Multi-State RNNscited by 120
  23. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
  24. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  25. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  26. 2021Finetuning Pretrained Transformers into RNNscited by 94
  27. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  28. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  29. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  30. 2023Transformers in Healthcare: A Surveycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 28; 2026: 18.

75 papers.

Sort:

By area:

  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  5. 2025MemOS: A Memory OS for AI Systemcited by 129
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  8. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  9. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  10. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  11. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 68
  12. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
38 more
  1. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  2. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  3. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  4. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  5. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  6. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  7. 2025Cognitive Memory in Large Language Modelscited by 28
  8. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  9. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  10. 2024A Tighter Complexity Analysis of SparseGPTcited by 19
  11. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  12. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  13. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  14. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
  15. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  16. 2024Towards LifeSpan Cognitive Systemscited by 13
  17. 2025CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Evictioncited by 13
  18. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  19. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  20. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  21. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  22. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8
  23. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMscited by 8
  24. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decodercited by 7
  25. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  26. 2025Scalable LLM Reasoning Acceleration with Low-rank Distillationcited by 7
  27. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  28. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  29. 2025Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correctioncited by 6
  30. 2026FASA: Frequency-aware Sparse Attentioncited by 6
  31. 2026MonarchRT: Efficient Attention for Real-Time Video Generationcited by 6
  32. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  33. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  34. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  35. 2024An Evolved Universal Transformer Memorycited by 4
  36. 2024EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importancecited by 4
  37. 2025AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Modelscited by 4
  38. 2026Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generationcited by 4

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  2. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  3. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  7. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference