# Gaussian Equivalence for Multi-Head Self-Attention

arXiv: https://arxiv.org/abs/2610.10033 · HTML: https://mlsanity.com/paper/2610.10033/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 10 of its 47 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [Effective Theory of Transformers at Initialization](https://mlsanity.com/paper/2304.02034/) (2023)
- [Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection](https://mlsanity.com/paper/2603.04427/) (2026)
- [On the Interpolation Error of Nonlinear Attention versus Linear Regression](https://mlsanity.com/paper/2506.18656/) (2025)

## Abstract (arXiv)

A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.