mlsanityData to 7 Oct 2026

D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models

arXiv 2406.13035 ↗First posted 18 Jun 2024cs.CLcited by 30

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Generative inference in Large Language Models (LLMs) is impeded by the growing memory demands of Key-Value (KV) cache, especially for longer sequences. Traditional KV cache eviction strategies, which discard less critical KV pairs based on attention scores, often degrade generation quality, leading to issues such as context loss or hallucinations. In this work, we introduce Dynamic Discriminative Operations (D2O), a KV cache compression method that optimizes KV cache size dynamically and discriminatively at two levels without fine-tuning, while preserving essential context. At layer level, D2O leverages the varying densities of attention weights between shallow and deep layers to dynamically determine which layers should avoid excessive eviction via a novel dynamic allocation strategy to minimize information loss. At token level, D2O incorporates a compensation mechanism that maintains a similarity threshold to re-discriminate the importance of currently discarded tokens, determining whether they should be recalled and merged with similar tokens. We conduct experiments on various benchmarks and LLM architectures. Our results show that D2O not only achieves significant memory savings and enhances inference throughput by more than 3× but also maintains high-quality long-text generation.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 58 of its 90 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

58 of 90 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024LoCoCo: Dropping In Convolutions for Long Context Compression8 Jun
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  3. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  6. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  7. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 48

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2017Graph Attention Networkscited by 8,854
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Mistral 7Bcited by 3,065
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
36 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2018CoQA: A Conversational Question Answering Challengecited by 674
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2023The Falcon Series of Open Language Modelscited by 526
  9. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  12. 2023Explainability for Large Language Models: A Surveycited by 331
  13. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  14. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  15. 2023Efficient Large Language Models: A Surveycited by 183
  16. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  17. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  18. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  19. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  20. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  21. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  22. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  23. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  24. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  25. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  26. 2023ETP: Learning Transferable ECG Representations via ECG-Text Pre-trainingcited by 28
  27. 2023How to Scale Your EMAcited by 22
  28. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  29. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  30. 2022Self-consistent Reasoning For Solving Math Word Problemscited by 10
  31. 2022G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Taskscited by 9
  32. 2022Spatio-Temporal Contrastive Learning Enhanced GNNs for Session-based Recommendationcited by 8
  33. 2024Structure-based Drug Design Benchmark: Do 3D Methods Really Dominate?cited by 7
  34. 2020Large Scale Many-Objective Optimization Driven by Distributional Adversarial Networkscited by 6
  35. 2020Many-Objective Estimation of Distribution Optimization Algorithm Based on WGAN-GPcited by 6
  36. 2023Text Classification: A Perspective of Deep Learning Methodscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 19.

30 papers.

Sort:

By area:

  1. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 72
  2. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  3. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 15
  4. 2025Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformerscited by 13
  5. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 9
  6. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  7. 2026STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstructioncited by 8
  8. 2025FreqKV: Key-Value Compression in Frequency Domain for Context Window Extensioncited by 7
  9. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMscited by 7
  10. 2026SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Mergingcited by 5
  11. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4
  12. 2026Dynamic Linear Attentioncited by 4
18 more
  1. 2026PRISM: Pareto-Efficient Retrieval over Intent-Aware Structured Memory for Long-Horizon Agentscited by 2
  2. 2026Information-Aware KV Cache Compression for Long Reasoningcited by 2
  3. 2026Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoningcited by 2
  4. 2025Neural Attention Searchcited by 1
  5. 2025Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimizationcited by 1
  6. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inferencecited by 1
  7. 2026Exploring a Layer-Wise Design Space for KV Cache Evictioncited by 1
  8. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inferencecited by 1
  9. 2025PaceLLM: Brain-Inspired Large Language Models for Long-Context Understandingcited by 0
  10. 2026Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inferencecited by 0
  11. 2026CASK: Core-Aware Selective KV Compression for Reasoning Tracescited by 0
  12. 2026Head-Aware Key-Value Compression for Efficient Autoregressive Image Generationcited by 0
  13. 2026Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compressioncited by 0
  14. 2026SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensationcited by 0
  15. 2026Beyond Sparse Weights: When Is Attention Compressible?cited by 0
  16. 2026Sigmoid Attention as a Better Substrate for Learned KV Cache Evictioncited by 0
  17. 2026Squeezing the Cache, Preserving the Truth: Monotonic Equipotential Allocation with Geodesia-KVcited by 0
  18. 2026ARM: Attention with Routed-Memory for Learnable Sparse Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  2. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  5. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
  6. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  7. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  8. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs