mlsanityData to 14 Oct 2026

HashAttention: Semantic Sparsity for Faster Inference

arXiv 2412.14468 ↗First posted 19 Dec 2024cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Leveraging long contexts is crucial for advanced AI systems, but attention computation poses a scalability challenge. While scaled dot-product attention (SDPA) exhibits token sparsity, i.e. only a few pivotal tokens significantly contribute to output, exploiting this sparsity remains challenging. Existing methods either suffer from quality degradation or require substantial additional resources. We show that identifying pivotal tokens is a Maximum Inner Product Search (MIPS) problem. However, existing MIPS solutions are not well-suited for SDPA, as they are not GPU-friendly and often underperform due to the separated query and key distributions. This paper introduces HashAttention, framing pivotal token identification as a recommendation problem. Given a query, HashAttention encodes keys and queries in Hamming space, capturing the required semantic similarity, using learned mapping functions. HashAttention efficiently identifies pivotal tokens for a given query using bitwise operations and computes attention using only these tokens, improving the overall attention efficiency. Trained on generic data, HashAttention reduces tokens used by up to 16× with minimal quality loss, requiring only 32 bits of auxiliary memory per token. Sparsity can be further improved to 32× through task-specific fine-tuning. On A100 GPU, at 32× sparsity, incorporating HashAttention reduces attention latency by up to 4.3× in GPT-FAST and 2.54× in FlashDecode, and achieves up to 3.12× higher throughput for GPT-FAST.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

34 of 45 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023Mistral 7Bcited by 3,164
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
22 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  8. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  9. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  10. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  11. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  12. 2021Dive into Deep Learningcited by 111
  13. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  14. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  15. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  16. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  17. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  18. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  19. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  20. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  21. 2021Attention Approximates Sparse Distributed Memorycited by 29
  22. 2014On Symmetric and Asymmetric LSHs for Inner Product Searchcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 15.

24 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  2. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 54
  3. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 14
  4. 2026Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsificationcited by 9
  5. 2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inferencecited by 7
  6. 2025Trainable Dynamic Mask Sparse Attentioncited by 6
  7. 2025vAttention: Verified Sparse Attentioncited by 6
  8. 2025Adaptive Computation Pruning for the Forgetting Transformercited by 5
  9. 2025Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attentioncited by 5
  10. 2026SOCKET: SOft Collision Kernel EsTimator for Sparse Attentioncited by 3
  11. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inferencecited by 2
  12. 2025Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategiescited by 1
12 more
  1. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systemscited by 1
  2. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cachecited by 1
  3. 2026Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biasescited by 1
  4. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decodingcited by 1
  5. 2026Inference Time Context Sparsity: Illusion or Opportunity?cited by 0
  6. 2026Training-Free Hashing-Based Attention via Binary Principal Componentscited by 0
  7. 2026Beyond Sparse Weights: When Is Attention Compressible?cited by 0
  8. 2026Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inferencecited by 0
  9. 2026CoWindow Attention: Full Causal Coverage Is a Collective Propertycited by 0
  10. 2026PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attentioncited by 0
  11. 2026The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendscited by 0
  12. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0