# Long-Context Generalization with Sparse Attention

arXiv: https://arxiv.org/abs/2506.16640 · HTML: https://mlsanity.com/paper/2506.16640/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 64 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Exploring Length Generalization in Large Language Models](https://mlsanity.com/paper/2207.04901/) (2022)
- [Zoology: Measuring and Improving Recall in Efficient Language Models](https://mlsanity.com/paper/2312.04927/) (2023)
- [What Algorithms can Transformers Learn? A Study in Length Generalization](https://mlsanity.com/paper/2310.16028/) (2023)
- [Sparse Sequence-to-Sequence Models](https://mlsanity.com/paper/1905.05702/) (2019)

## Built on by (most cited first, 21 of 21)

- [Addressable Memory for Video World Models](https://mlsanity.com/paper/2608.07408/) (2026)
- [DGP: A Dual-Granularity Prompting Framework for Fraud Detection with Graph-Enhanced LLMs](https://mlsanity.com/paper/2507.21653/) (2025)
- [Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling](https://mlsanity.com/paper/2601.12145/) (2026)
- [NOSA: Native and Offloadable Sparse Attention](https://mlsanity.com/paper/2510.13602/) (2025)
- [Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs](https://mlsanity.com/paper/2605.00814/) (2026)
- [DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention](https://mlsanity.com/paper/2605.18753/) (2026)
- [Controllably Efficient Language Models](https://mlsanity.com/paper/2511.05313/) (2025)
- [AdaSplash-2: Faster Differentiable Sparse Attention](https://mlsanity.com/paper/2604.15180/) (2026)
- [Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale](https://mlsanity.com/paper/2607.01538/) (2026)
- [Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration](https://mlsanity.com/paper/2509.21848/) (2025)
- [Context Cartography: Toward Structured Governance of Contextual Space in Large Language Model Systems](https://mlsanity.com/paper/2603.20578/) (2026)
- [Screening Is Enough](https://mlsanity.com/paper/2604.01178/) (2026)
- [ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory](https://mlsanity.com/paper/2606.25156/) (2026)
- [Perspectives on Tsallis Statistics for Artificial Intelligence](https://mlsanity.com/paper/2608.01223/) (2026)
- [An empirical study on the limitation of Transformers in program trace generation](https://mlsanity.com/paper/2509.25073/) (2025)
- [Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding](https://mlsanity.com/paper/2604.24820/) (2026)
- [EntmaxKV: Support-Aware Decoding for Entmax Attention](https://mlsanity.com/paper/2605.21649/) (2026)
- [Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D](https://mlsanity.com/paper/2607.16072/) (2026)
- [HHR: Hierarchical Hash Retrieval for Efficient LLM Generation](https://mlsanity.com/paper/2610.01230/) (2026)
- [Length Generalization Needs Proper Regularization](https://mlsanity.com/paper/2610.04518/) (2026)
- [FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation](https://mlsanity.com/paper/2610.17191/) (2026)

## Abstract (arXiv)

Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many settings, this density has been shown to be detrimental for tasks that demand precise focus on fixed-size patterns: as sequence length increases, non-informative tokens accumulate attention probability mass, leading to dispersion and representational collapse. We show in this paper that dynamically sparse attention mechanisms using α-entmax can avoid these issues, due to their ability to assign exact zeros to irrelevant tokens. Furthermore, we introduce Adaptive-Scalable Entmax (ASEntmax), which endows α-entmax with a learnable temperature parameter, allowing the attention distribution to interpolate between sparse (pattern-focused) and dense (softmax-like) regimes. Our empirical evaluation on synthetic tasks and language modeling demonstrates that ASEntmax substantially outperforms softmax, scalable softmax, and fixed-temperature α-entmax baselines, achieving up to 1000× length extrapolation on synthetic benchmarks and superior long-context generalization on language modeling while preserving short-context performance, including better perplexity trends and higher retrieval accuracies at 8× training length.