mlsanityData to 7 Oct 2026

Mapping of attention mechanisms to a generalized Potts model

arXiv 2304.07235 ↗First posted 14 Apr 2023cond-mat.dis-nn cond-mat.stat-mech cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we show analytically that if one decouples the treatment of word positions and embeddings, a single layer of self-attention learns the conditionals of a generalized Potts model with interactions between sites and Potts colors. Moreover, we show that training this neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method, well known in statistical physics. Using this mapping, we compute the generalization error of self-attention in a model scenario analytically using the replica method.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 20 of its 56 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

20 of 56 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  7. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  8. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  9. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  10. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  11. 2017Structured Attention Networkscited by 95
  12. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
8 more
  1. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  2. 2017Inverse statistical problems: from the inverse Ising problem to data sciencecited by 36
  3. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  4. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  5. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  6. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  7. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  8. 2022The impact of memory on learning sequence-to-sequence taskscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 7; 2026: 2.

19 papers.

Sort:

  1. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  2. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  3. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  4. 2024Anchor function: a type of benchmark functions for studying language modelscited by 10
  5. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  6. 2023Eight challenges in developing theory of intelligencecited by 6
  7. 2024Are queries and keys always relevant? A case study on Transformer wave functionscited by 5
  8. 2024Dynamical Mean-Field Theory of Self-Attention Neural Networkscited by 3
  9. 2024Self-attention as an attractor network: transient memories without backpropagationcited by 3
  10. 2024Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokenscited by 3
  11. 2025Capturing AI's Attention: Physics of Repetition, Hallucination, Bias and Beyondcited by 3
  12. 2025Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplingscited by 2
7 more
  1. 2025Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problemscited by 1
  2. 2025Self-attention vector output similarities reveal how machines pay attentioncited by 1
  3. 2025Multispin Physics of AI Tipping Points and Hallucinationscited by 0
  4. 2025Machine learning approaches for interpretable antibody property prediction using structural datacited by 0
  5. 2025Testing Transformer Learnability on the Arithmetic Sequence of Rooted Treescited by 0
  6. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
  7. 2026Sampling at intermediate temperatures is optimal for training large language models in protein structure predictioncited by 0