# Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics

arXiv: https://arxiv.org/abs/2605.29351 · HTML: https://mlsanity.com/paper/2605.29351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 57 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 7 of 7)

- [Attention Residuals](https://mlsanity.com/paper/2603.15031/) (2026)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Back to Basics: Let Denoising Generative Models Denoise](https://mlsanity.com/paper/2511.13720/) (2025)
- [Solving Empirical Bayes via Transformers](https://mlsanity.com/paper/2502.09844/) (2025)
- [DiScoFormer: Plug-In Density and Score Estimation with Transformers](https://mlsanity.com/paper/2511.05924/) (2025)
- [Generative Modeling from Black-box Corruptions via Self-Consistent Stochastic Interpolants](https://mlsanity.com/paper/2512.10857/) (2025)
- [Constrained Denoising, Empirical Bayes, and Optimal Transport](https://mlsanity.com/paper/2506.09986/) (2025)

## Abstract (arXiv)

We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a depth-dependent energy landscape governing in-context inference. We show that effective denoising can emerge without an explicit noise schedule: a fixed kernel bandwidth and finite integration horizon suffice, yielding a principled depth-noise relationship. We further establish a posterior-mean recovery guarantee for a class of well-behaved priors, where the empirical estimator converges to the Bayes-optimal predictor under asymptotic conditions. Connecting these dynamics to reverse-diffusion limits, our results provide a statistical interpretation of attention as in-context inference via sample-based posterior estimation, without explicit density modeling.