# The Bayesian Geometry of Transformer Attention

arXiv: https://arxiv.org/abs/2512.22471 · HTML: https://mlsanity.com/paper/2512.22471/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 27 Dec 2025.

## Builds on (most cited first, 12 of 12)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024)
- [In-Context Learning through the Bayesian Prism](https://mlsanity.com/paper/2306.04891/) (2023)
- [Can Transformers Learn Full Bayesian Inference in Context?](https://mlsanity.com/paper/2501.16825/) (2025)

## Built on by (most cited first, 13 of 13)

- [Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking](https://mlsanity.com/paper/2602.10743/) (2026)
- [The Geometric Reasoner: Manifold-Informed Latent Foresight Search for Long-Context Reasoning](https://mlsanity.com/paper/2601.18832/) (2026)
- [Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models](https://mlsanity.com/paper/2602.08658/) (2026)
- [Position: agentic AI orchestration should be Bayes-consistent](https://mlsanity.com/paper/2605.00742/) (2026)
- [Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability](https://mlsanity.com/paper/2608.19338/) (2026)
- [Gradient Descent as Implicit EM in Distance-Based Neural Models](https://mlsanity.com/paper/2512.24780/) (2025)
- [Universal priors: solving empirical Bayes via Bayesian inference and pretraining](https://mlsanity.com/paper/2602.15136/) (2026)
- [ObserverBench: Testing Mechanistic Estimates for Intervention and Control](https://mlsanity.com/paper/2609.03026/) (2026)
- [Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination](https://mlsanity.com/paper/2605.05686/) (2026)
- [In-context learning enables continental-scale subsurface temperature prediction from sparse local observations](https://mlsanity.com/paper/2605.16665/) (2026)
- [Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference](https://mlsanity.com/paper/2605.28384/) (2026)
- [Bayesian Wind Tunnels for Model Selection](https://mlsanity.com/paper/2607.19379/) (2026)
- [Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanism](https://mlsanity.com/paper/2609.35161/) (2026)

## Abstract (arXiv)

Transformers often appear to perform Bayesian reasoning in context, but verifying this rigorously has been impossible: natural data lack analytic posteriors, and large models conflate reasoning with memorization. We address this by constructing Bayesian wind tunnels -- controlled environments where the true posterior is known in closed form and memorization is provably impossible. In these settings, small transformers reproduce Bayesian posteriors with 10-3-10-4 bit accuracy, while capacity-matched MLPs fail by orders of magnitude, establishing a clear architectural separation. Across two tasks -- bijection elimination and Hidden Markov Model (HMM) state tracking -- we find that transformers implement Bayesian inference through a consistent geometric mechanism: residual streams serve as the belief substrate, feed-forward networks perform the posterior update, and attention provides content-addressable routing. Geometric diagnostics reveal orthogonal key bases, progressive query-key alignment, and a low-dimensional value manifold parameterized by posterior entropy. During training this manifold unfurls while attention patterns remain stable, a frame-precision dissociation predicted by recent gradient analyses. Taken together, these results demonstrate that hierarchical attention realizes Bayesian inference by geometric design, explaining both the necessity of attention and the failure of flat architectures. Bayesian wind tunnels provide a foundation for mechanistically connecting small, verifiable systems to reasoning phenomena observed in large language models.