mlsanityData to 14 Oct 2026

Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification

arXiv 2502.09647 ↗First posted 11 Feb 2025cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The ability to process long contexts is crucial for many natural language processing tasks, yet it remains a significant challenge. While substantial progress has been made in enhancing the efficiency of attention mechanisms, there is still a gap in understanding how attention heads function in long-context settings. In this paper, we observe that while certain heads consistently attend to local information only, others swing between attending to local and long-context information depending on the query. This raises the question: can we identify which heads require long-context information to predict the next token accurately? We demonstrate that it's possible to predict which heads are crucial for long-context processing using only local keys. The core idea here is to exploit a simple model for the long-context scores via second moment approximations. These findings unveil simple properties of attention in the context of long sequences, and open the door to potentially significant gains in efficiency.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 37 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

34 of 37 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023Mistral 7Bcited by 3,164
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
22 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  7. 2024Massive Activations in Large Language Modelscited by 294
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  9. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  10. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  11. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  12. 2023Efficient Large Language Models: A Surveycited by 186
  13. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  14. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  15. 2024Transformers are Multi-State RNNscited by 120
  16. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  17. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  18. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  19. 2024Attention Heads of Large Language Models: A Surveycited by 67
  20. 2024ShadowLLM: Predictor-based Contextual Sparsity for Large Language Modelscited by 13
  21. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  22. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Headscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 21
  2. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  3. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decodingcited by 3
  4. 2025Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformerscited by 0
  5. 2026BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMscited by 0