# Spectral alignment of stochastic gradient descent for high-dimensional classification tasks

arXiv: https://arxiv.org/abs/2310.03010 · HTML: https://mlsanity.com/paper/2310.03010/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2023.

## Built on by (most cited first, 13 of 13)

- [Linguistic Collapse: Neural Collapse in (Large) Language Models](https://mlsanity.com/paper/2405.17767/) (2024)
- [Does SGD really happen in tiny subspaces?](https://mlsanity.com/paper/2405.16002/) (2024)
- [Nonlinear spiked covariance matrices and signal propagation in deep neural networks](https://mlsanity.com/paper/2402.10127/) (2024)
- [Efficient Sketches for Training Data Attribution and Studying the Loss Landscape](https://mlsanity.com/paper/2402.03994/) (2024)
- [Feature learning as alignment: a structural property of gradient descent in non-linear neural networks](https://mlsanity.com/paper/2402.05271/) (2024)
- [Transformer Block Coupling and its Correlation with Generalization in LLMs](https://mlsanity.com/paper/2407.07810/) (2024)
- [Deconstructing the Goldilocks Zone of Neural Network Initialization](https://mlsanity.com/paper/2402.03579/) (2024)
- [Universality of high-dimensional scaling limits of stochastic gradient descent](https://mlsanity.com/paper/2512.13634/) (2025)
- [Spectral phase transitions and trainability in neural network learning dynamics](https://mlsanity.com/paper/2606.28486/) (2026)
- [Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey](https://mlsanity.com/paper/2410.12927/) (2024)
- [Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generation](https://mlsanity.com/paper/2510.17120/) (2025)
- [A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights](https://mlsanity.com/paper/2605.16913/) (2026)
- [Boosting Data Augmentation with Stochastic Weight Averaging](https://mlsanity.com/paper/2608.14373/) (2026)

## Abstract (arXiv)

We rigorously study the relation between the training dynamics via stochastic gradient descent (SGD) and the spectra of empirical Hessian and gradient matrices. We prove that in two canonical classification tasks for multi-class high-dimensional mixtures and either 1 or 2-layer neural networks, both the SGD trajectory and emergent outlier eigenspaces of the Hessian and gradient matrices align with a common low-dimensional subspace. Moreover, in multi-layer settings this alignment occurs per layer, with the final layer's outlier eigenspace evolving over the course of training, and exhibiting rank deficiency when the SGD converges to sub-optimal classifiers. This establishes some of the rich predictions that have arisen from extensive numerical studies in the last decade about the spectra of Hessian and information matrices over the course of training in overparametrized networks.