# Transformers as Measure-Theoretic Associative Memory: A Statistical Perspective and Minimax Optimality

arXiv: https://arxiv.org/abs/2602.01863 · HTML: https://mlsanity.com/paper/2602.01863/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2026.

## Built on by (most cited first, 4 of 4)

- [One Operator for Many Densities: Amortized Approximation of Conditioning by Neural Operators](https://mlsanity.com/paper/2605.06873/) (2026)
- [Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learning](https://mlsanity.com/paper/2605.05189/) (2026)
- [Propagation of Chaos in Contextual Flow Maps](https://mlsanity.com/paper/2605.16747/) (2026)
- [A Unified Framework for In-Context Learning with Causal and Masked Language Models](https://mlsanity.com/paper/2607.04081/) (2026)

## Abstract (arXiv)

Transformers excel through content-addressable retrieval and the ability to exploit contexts of, in principle, unbounded length. We recast associative memory at the level of probability measures, treating a context as a distribution over tokens and viewing attention as an integral operator on measures. Concretely, for mixture contexts nu = I-1 sumi=1I μ(i^*) and a query xq(i^*), the task decomposes into (i) recall of the relevant component μ(i^*) and (ii) prediction from (μi^*,xq). We study learned softmax attention (not a frozen kernel) trained by empirical risk minimization and show that a shallow measure-theoretic Transformer composed with an MLP learns the recall-and-predict map under a spectral assumption on the input densities. We further establish a matching minimax lower bound with the same rate exponent (up to multiplicative constants), proving sharpness of the convergence order. The framework offers a principled recipe for designing and analyzing Transformers that recall from arbitrarily long, distributional contexts with provable generalization guarantees.