mlsanityData to 7 Oct 2026

HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inference

arXiv 2402.09360 ↗First posted 14 Feb 2024cs.LG cs.AIcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Autoregressive decoding with generative Large Language Models (LLMs) on accelerators (GPUs/TPUs) is often memory-bound where most of the time is spent on transferring model parameters from high bandwidth memory (HBM) to cache. On the other hand, recent works show that LLMs can maintain quality with significant sparsity/redundancy in the feedforward (FFN) layers by appropriately training the model to operate on a top-k fraction of rows/columns (where k ≈ 0.05), there by suggesting a way to reduce the transfer of model parameters, and hence latency. However, exploiting this sparsity for improving latency is hindered by the fact that identifying top rows/columns is data-dependent and is usually performed using full matrix operations, severely limiting potential gains. To address these issues, we introduce HiRE (High Recall Approximate Top-k Estimation). HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-k rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-k: an efficient multi-device approximate top-k operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by 1.47× on a single TPUv5e device.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 59 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

43 of 59 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
31 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  9. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  10. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  11. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  13. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  14. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  15. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  16. 2020Sparse GPU Kernels for Deep Learningcited by 138
  17. 2023Controlled Decoding from Language Modelscited by 133
  18. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  19. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  20. 2021Sparse is Enough in Scaling Transformerscited by 86
  21. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  22. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  23. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  24. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  25. 2023Intriguing Properties of Quantization at Scalecited by 48
  26. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  27. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  28. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 15
  29. 2023Alternating Updates for Efficient Transformerscited by 13
  30. 2023Accelerating Deep Neural Networks via Semi-Structured Activation Sparsitycited by 10
  31. 2022Treeformer: Dense Gradient Trees for Efficient Attention Computationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 1.

7 papers.

  1. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
  2. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  3. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  4. 2024MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layerscited by 4
  5. 2024Scaling the Vocabulary of Non-autoregressive Models for Efficient Generative Retrievalcited by 2
  6. 2026Stochastic Sparse Attention for Memory-Bound Inferencecited by 1
  7. 2025A Faster Generalized Two-Stage Approximate Top-Kcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  3. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  4. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  5. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  6. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  8. 2025Draft-based Approximate Inference for LLMs