# High-Dimensional Learning Dynamics of Attention-Indexed Models

arXiv: https://arxiv.org/abs/2609.03858 · HTML: https://mlsanity.com/paper/2609.03858/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 38 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 5 of 5)

- [Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime](https://mlsanity.com/paper/2509.24882/) (2025)
- [Statistical Advantage of Softmax Attention: Insights from Single-Location Regression](https://mlsanity.com/paper/2509.21936/) (2025)
- [High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks](https://mlsanity.com/paper/2601.10483/) (2026)
- [Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networks](https://mlsanity.com/paper/2604.03068/) (2026)
- [Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws](https://mlsanity.com/paper/2608.13335/) (2026)

## Abstract (arXiv)

Attention mechanisms are central to modern foundation models, yet their training dynamics remain poorly understood, especially when the attention matrices have extensive rank. In this work, we study attention-indexed models, a broad framework that can represent multi-layer and multi-head attention architectures. First, we show that, in a suitable high-dimensional limit, the population-loss landscape is characterized by a finite set of trace order parameters. In contrast, online stochastic gradient descent (SGD) is governed by an infinite hierarchy of matrix moments, which we show can be exponentially well-approximated by a finite truncated system. Second, this framework reveals that attention parameterization itself can act as an architectural implicit bias. Direct optimization of an attention matrix S∈ℝd× d can remain trapped in an uninformative state. Tied attention (S=WW^top) induces an automatic symmetry-breaking mechanism and yields weak recovery in Theta(d2log d) samples. For untied attention, S=UV^top, we uncover a fast-slow mechanism: the pre-activation mean first evolves on a fast timescale, while the overlaps evolve on a slower one. Weak recovery on the Theta(d2log d) scale occurs when the state selected by the fast dynamics breaks the initial symmetry.