mlsanityData to 14 Oct 2026

Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation

arXiv 2505.22842 ↗First posted 28 May 2025cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Transformer-based language models rely on positional encoding (PE) to handle token order and support context length extrapolation. However, existing PE methods lack theoretical clarity and rely on limited evaluation metrics to substantiate their extrapolation claims. We propose the Bayesian Attention Mechanism (BAM), a theoretical framework that formulates positional encoding as a prior within a probabilistic model. BAM unifies existing methods (e.g., NoPE and ALiBi) and motivates a new Generalized Gaussian positional prior that substantially improves long-context generalization. Empirically, BAM enables accurate information retrieval at 500× the training context length, outperforming previous state-of-the-art context length generalization in long context retrieval accuracy while maintaining comparable perplexity and introducing minimal additional parameters.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 24 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

23 of 24 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2023Mistral 7Bcited by 3,164
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
11 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  4. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  5. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  6. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  7. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  8. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  9. 2025Scalable-Softmax Is Superior for Attentioncited by 45
  10. 2023Attention: Marginal Probability is All You Need?cited by 4
  11. 2025Computation Mechanism Behind LLM Position Generalizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Robust Filter Attention: Self-Attention as Precision-Weighted State Estimationcited by 1
  2. 2026The Transformer as a Polar State Estimatorcited by 0
  3. 2026A Probabilistic Interpretation of KV Cache Evictioncited by 0