mlsanityData to 7 Oct 2026

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

arXiv 2306.14048 ↗First posted 24 Jun 2023cs.LGcited by 805

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs), despite their recent impressive accomplishments, are notably cost-prohibitive to deploy, particularly for applications involving long-content generation, such as dialogue systems and story writing. Often, a large amount of transient state information, referred to as the KV cache, is stored in GPU memory in addition to model parameters, scaling linearly with the sequence length and batch size. In this paper, we introduce a novel approach for implementing the KV cache which significantly reduces its memory footprint. Our approach is based on the noteworthy observation that a small portion of tokens contributes most of the value when computing attention scores. We call these tokens Heavy Hitters (H2). Through a comprehensive investigation, we find that (i) the emergence of H2 is natural and strongly correlates with the frequent co-occurrence of tokens in the text, and (ii) removing them results in significant performance degradation. Based on these insights, we propose Heavy Hitter Oracle (H2O), a KV cache eviction policy that dynamically retains a balance of recent and H2 tokens. We formulate the KV cache eviction as a dynamic submodular problem and prove (under mild assumptions) a theoretical guarantee for our novel eviction algorithm which could help guide future work. We validate the accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of tasks. Our implementation of H2O with 20% heavy hitters improves the throughput over three leading inference systems DeepSpeed Zero-Inference, Hugging Face Accelerate, and FlexGen by up to 29×, 29×, and 3× on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the latency by up to 1.9×. The code is available at https://github.com/FMInference/H2O.

Key facts. Cited by 805 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 98 of its 154 references that are in the map. First posted on arXiv 24 Jun 2023.

Builds on

98 of 154 references.

References · 98

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  12. 2020Reformer: The Efficient Transformercited by 1,658
  13. 2020Rethinking Attention with Performerscited by 1,584
  14. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  15. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  16. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  17. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  18. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  19. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  20. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  21. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  22. 2020Efficient Transformers: A Surveycited by 784
  23. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  24. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  25. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  26. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  27. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  28. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  29. 2018Rethinking the Value of Network Pruningcited by 595
  30. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  31. 2022Efficiently Scaling Transformer Inferencecited by 529
  32. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  33. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  34. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  35. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  36. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  37. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  38. 2023Learning to Compress Prompts with Gist Tokenscited by 379

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 142; 2025: 256; 2026: 397.

805 papers.

Sort:

By area:

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  4. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  6. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  8. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  9. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  10. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  11. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
38 more
  1. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  4. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 169
  7. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  8. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 160
  9. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  10. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  11. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  12. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  13. 2025MemOS: A Memory OS for AI Systemcited by 129
  14. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  15. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  16. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  17. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  18. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  19. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  20. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  21. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  22. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  23. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  24. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  25. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  26. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  27. 2024Long Context Compression with Activation Beaconcited by 101
  28. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  29. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  30. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  31. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  32. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  33. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  34. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  35. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  36. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  37. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  38. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89

50 most cited of 805 All 805 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  2. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughput
  3. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  6. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
  7. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization