mlsanityData to 14 Oct 2026

Scale-invariant Attention

arXiv 2505.17083 ↗First posted 20 May 2025cs.CL cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

One persistent challenge in LLM research is the development of attention mechanisms that are able to generalise from training on shorter contexts to inference on longer contexts. We propose two conditions that we expect all effective long context attention mechanisms to have: scale-invariant total attention, and scale-invariant attention sparsity. Under a Gaussian assumption, we show that a simple position-dependent transformation of the attention logits is sufficient for these conditions to hold. Experimentally we find that the resulting scale-invariant attention scheme gives considerable benefits in terms of validation loss when zero-shot generalising from training on short contexts to validation on longer contexts, and is effective at long-context retrieval.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

39 of 58 references.

References · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2025Gemma 3 Technical Reportcited by 1,824
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
27 more
  1. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  2. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  3. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  4. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  6. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  7. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  8. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  9. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  10. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  11. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  12. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  13. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  14. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  15. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  16. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  17. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  18. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  19. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  20. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  21. 2025Command A: An Enterprise-Ready Large Language Modelcited by 59
  22. 2025Scalable-Softmax Is Superior for Attentioncited by 45
  23. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 27
  24. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  25. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  26. 2025EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devicescited by 5
  27. 2025Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanismscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Long-Context Generalization with Sparse Attentioncited by 21
  2. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  3. 2026AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generationcited by 6
  4. 2026Scaling Limits of Long-Context Transformerscited by 3
  5. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0