mlsanityData to 7 Oct 2026

Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices

arXiv 2410.01805 ↗First posted 2 Oct 2024cs.CLcited by 15

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Scaling the input context length of a large language model (LLM) incurs a significant increase in computation cost and memory footprint to maintain the attention key-value (KV) cache. Existing KV cache compression methods suffer from inefficient compression strategies and limited memory reduction effects, making it difficult for LLMs to conduct long-context inference on consumer-grade devices, especially when inferring long-context stream input. Such obstacles prevent consumer-grade devices from supporting more complex applications, creating challenges for the democratization of LLMs. To overcome this, we propose Locret, the first framework to create an eviction policy compatible with chunked prefill. By evaluating the causal importance of KV cache units by learnable retaining heads, Locret enables precise eviction of cache units, facilitating efficient long-context inference. In our extensive empirical studies, Locret outperforms the recent popular and competitive approaches in terms of memory efficiency and generation quality -- Locret achieves up to 20x of KV cache compression ratio within less than 10% performance loss. Furthermore, Locret achieves 128K+ long-context inference on a single NVIDIA 4090 GPU without compromising generation quality and only costs <1 GPU hour of additional training.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 62 of its 85 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

62 of 85 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Inference-Friendly Models With MixAttention23 Sep
  2. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  3. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024LoCoCo: Dropping In Convolutions for Long Context Compression8 Jun
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  7. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  10. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
1 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 49

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
37 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  11. 2024Large Language Models: A Surveycited by 506
  12. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  13. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  14. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  15. 2021Efficient Attentions for Long Document Summarizationcited by 342
  16. 2023Tool Learning with Foundation Modelscited by 302
  17. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  18. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  19. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  20. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  21. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  22. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  23. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  24. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  25. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  26. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  27. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  28. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  29. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  30. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  31. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  32. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  33. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  34. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  35. 2024Making Long-Context Language Models Better Multi-Hop Reasonerscited by 25
  36. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  37. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  2. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  3. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  4. 2026Learning to Evict from Key-Value Cachecited by 12
  5. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  6. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  7. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  8. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  9. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMscited by 4
  10. 2025APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUscited by 3
  11. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  12. 2025Can Large Language Models Understand As Well As Apply Patent Regulations to Pass a Hands-On Patent Attorney Test?cited by 0
3 more
  1. 2026Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attentioncited by 0
  2. 2026Discovering KV Cache Eviction Policies via LLM-Guided Program Evolutioncited by 0
  3. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cachecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  3. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
  4. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
  5. 2024In-context KV-Cache Eviction for LLMs via Attention-Gate
  6. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
  8. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference