# Early-stopping for Transformer model training

arXiv: https://arxiv.org/abs/2510.16074 · HTML: https://mlsanity.com/paper/2510.16074/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Early Stopping in Deep Networks: Double Descent and How to Eliminate it](https://mlsanity.com/paper/2007.10099/) (2020)
- [Sequence Length is a Domain: Length-based Overfitting in Transformer Models](https://mlsanity.com/paper/2109.07276/) (2021)
- [Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models](https://mlsanity.com/paper/2505.16959/) (2025)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data](https://mlsanity.com/paper/2202.02842/) (2022)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Early Stopping Tabular In-Context Learning](https://mlsanity.com/paper/2506.21387/) (2025)
- [Unraveling the Gradient Descent Dynamics of Transformers](https://mlsanity.com/paper/2411.07538/) (2024)
- [Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression](https://mlsanity.com/paper/2502.13283/) (2025)
- [Grokking and Generalization Collapse: Insights from textttHTSR theory](https://mlsanity.com/paper/2506.04434/) (2025)
- [Enhancing Accuracy in Deep Learning Using Random Matrix Theory](https://mlsanity.com/paper/2310.03165/) (2023)
- [Pruning Deep Neural Networks via a Combination of the Marchenko-Pastur Distribution and Regularization](https://mlsanity.com/paper/2503.01922/) (2025)
- [Key-Value Transformer](https://mlsanity.com/paper/2305.19129/) (2023)

## Built on by (most cited first, 1 of 1)

- [The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry](https://mlsanity.com/paper/2604.22778/) (2026)

## Abstract (arXiv)

This work, based on Random Matrix Theory (RMT), introduces a novel early-stopping strategy for Transformer training dynamics. Utilizing the Power Law (PL) fit to tansformer attention matrices as a probe, we demarcate training into three stages: structural exploration, heavy-tailed structure stabilization, and convergence saturation. Empirically, we observe that the spectral density of the shallow self-attention matrix V consistently evolves into a heavy-tailed distribution. Crucially, we propose two consistent and validation-set-free criteria: a quantitative metric for heavy-tailed dynamics and a novel spectral signature indicative of convergence. The strong alignment between these criteria highlights the utility of RMT for monitoring and diagnosing the progression of Transformer model training.