# Attention to Order: Transformers Discover Phase Transitions via Learnability

arXiv: https://arxiv.org/abs/2510.07401 · HTML: https://mlsanity.com/paper/2510.07401/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Why does deep and cheap learning work so well?](https://mlsanity.com/paper/1608.08225/) (2016)
- [Discovering Phase Transitions with Unsupervised Learning](https://mlsanity.com/paper/1606.00318/) (2016)
- [Unsupervised learning of phase transitions: from principal component analysis to variational autoencoders](https://mlsanity.com/paper/1703.02435/) (2017)
- [A phase transition between positional and semantic learning in a solvable model of dot-product attention](https://mlsanity.com/paper/2402.03902/) (2024)
- [Quantum Many-Body Physics Calculations with Large Language Models](https://mlsanity.com/paper/2403.03154/) (2024)
- [Deep Learning the Ising Model Near Criticality](https://mlsanity.com/paper/1708.04622/) (2017)
- [Thermodynamics-inspired Explanations of Artificial Intelligence](https://mlsanity.com/paper/2206.13475/) (2022)
- [Unsupervised machine learning of quantum phase transitions using diffusion maps](https://mlsanity.com/paper/2003.07399/) (2020)
- [Neural Networks as Spin Models: From Glass to Hidden Order Through Training](https://mlsanity.com/paper/2408.06421/) (2024)

## Abstract (arXiv)

Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce learnability as a universal criterion, defined as the ability of a transformer model containing attention mechanism to extract structure from microscopic states. Using self-supervised learning and Monte Carlo generated configurations of the two-dimensional Ising model, we show that ordered phases correspond to enhanced learnability, manifested in both reduced training loss and structured attention patterns, while disordered phases remain resistant to learning. Two unsupervised diagnostics, the sharp jump in training loss and the rise in attention entropy, recover the critical temperature in excellent agreement with the exact value. Our results establish learnability as a data-driven marker of phase transitions and highlight deep parallels between long-range order in condensed matter and the emergence of structure in modern language models.