# Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

arXiv: https://arxiv.org/abs/2606.30226 · HTML: https://mlsanity.com/paper/2606.30226/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length](https://mlsanity.com/paper/1807.05031/) (2018)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [Emergent properties of the local geometry of neural loss landscapes](https://mlsanity.com/paper/1910.05929/) (2019)
- [Comparing Dynamics: Deep Neural Networks versus Glassy Systems](https://mlsanity.com/paper/1803.06969/) (2018)
- [Does SGD really happen in tiny subspaces?](https://mlsanity.com/paper/2405.16002/) (2024)
- [Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD](https://mlsanity.com/paper/2412.20553/) (2024)
- [SAM operates far from home: eigenvalue regularization as a dynamical phenomenon](https://mlsanity.com/paper/2302.08692/) (2023)
- [Negative eigenvalues of the Hessian in deep neural networks](https://mlsanity.com/paper/1902.02366/) (2019)
- [Beyond Random Matrix Theory for Deep Networks](https://mlsanity.com/paper/2006.07721/) (2020)
- [The Role of the Time-Dependent Hessian in High-Dimensional Optimization](https://mlsanity.com/paper/2403.02418/) (2024)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)

## Built on by (most cited first, 1 of 1)

- [Traversing the solution space of neural networks with Hessian Null Space Continuation](https://mlsanity.com/paper/2609.38081/) (2026)

## Abstract (arXiv)

Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors identify which parameters generate that curvature. In this work, we study how the leading Hessian eigenvectors evolve during training and how they affect the learning trajectories. We track the training dynamics of multilayer perceptrons on a classification problem and measure eigenvector dynamics through two complementary statistics: (i) displacement over time, inspired by analyses of glassy systems, and (ii) localization via the inverse participation ratio. The metrics are compared against a random null model of the Hessian induced by the architecture. Our results reveal clear optimizer-dependent behaviour. SGD leads to progressively more stable leading curvature directions, while Adam exhibits substantially stronger reorganization of eigenvectors throughout training. We also observe a localization phenomenon under Adam, where a small subset of parameters contributes disproportionately to the leading curvature directions. These results suggest that Hessian eigenvector dynamics capture key differences in optimizer behaviour and the resulting training trajectories.