mlsanityData to 7 Oct 2026

Does SGD really happen in tiny subspaces?

arXiv 2405.16002 ↗First posted 25 May 2024cs.LG math.OC stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

Understanding the training dynamics of deep neural networks is challenging due to their high-dimensional nature and intricate loss landscapes. Recent studies have revealed that, along the training trajectory, the gradient approximately aligns with a low-rank top eigenspace of the training loss Hessian, referred to as the dominant subspace. Given this alignment, this paper explores whether neural networks can be trained within the dominant subspace, which, if feasible, could lead to more efficient training methods. Our primary observation is that when the SGD update is projected onto the dominant subspace, the training loss does not decrease further. This suggests that the observed alignment between the gradient and the dominant subspace is spurious. Surprisingly, projecting out the dominant subspace proves to be just as effective as the original update, despite removing the majority of the original update component. We observe similar behavior across practical setups, including the large learning rate regime (also known as Edge of Stability), Sharpness-Aware Minimization, momentum, and adaptive optimizers. We discuss the main causes and implications of this spurious alignment, shedding light on the dynamics of neural network training.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 56 of its 73 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

56 of 73 references.

References · 56

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  9. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  10. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  11. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  12. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
38 more
  1. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  2. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  3. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  4. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  5. 2020Momentum Improves Normalized SGDcited by 136
  6. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  7. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  8. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  9. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  10. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  11. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  12. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  13. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  14. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  15. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  16. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  17. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  18. 2022Understanding the unstable convergence of gradient descentcited by 84
  19. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  20. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  21. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  22. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  23. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  24. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  25. 2022Learning threshold neurons via the "edge of stability"cited by 50
  26. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 46
  27. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  28. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  29. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  30. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  31. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  32. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 23
  33. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  34. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  35. 2020Improving Neural Network Training in Low Dimensional Random Basescited by 19
  36. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
  37. 2021Privately Learning Subspacescited by 16
  38. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 19.

35 papers.

Sort:

  1. 2025Dion: Distributed Orthonormalized Updatescited by 64
  2. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  3. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  4. 2025Hidden Breakthroughs in Language Model Trainingcited by 15
  5. 2026Insights on Muon from Simple Quadraticscited by 11
  6. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 8
  7. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  8. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  9. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 6
  10. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 6
  11. 2026Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minimacited by 4
  12. 2024Understanding Gradient Descent through the Training Jacobiancited by 3
23 more
  1. 2025BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Trainingcited by 3
  2. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 3
  3. 2026First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformerscited by 3
  4. 2026Too Sharp, Too Sure: When Calibration Follows Curvaturecited by 2
  5. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 2
  6. 2026Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Trainingcited by 2
  7. 2025Accelerating Neural Network Training Along Sharp and Flat Directionscited by 1
  8. 2025DOME: Improving Signal-to-Noise in Stochastic Gradient Descent via Sharp-Direction Subspace Filteringcited by 1
  9. 2025A Unified Noise-Curvature View of Loss of Trainabilitycited by 1
  10. 2025Geometrically Principled Randomized Optimization for Efficient LLM Trainingcited by 1
  11. 2026Depth, Not Data: An Analysis of Hessian Spectral Bifurcationcited by 1
  12. 2026Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuationscited by 1
  13. 2024Subspace Langevin Monte Carlocited by 0
  14. 2025Towards Scalable Backpropagation-Free Gradient Estimationcited by 0
  15. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  16. 2026Decentralized SGD with Controlled Disagreement Finds Flatter Minimacited by 0
  17. 2026Worker Disagreement Reveals Sharp Directions in Local SGDcited by 0
  18. 2026Rethinking Bregman Divergences in Kronecker-Factored Optimizerscited by 0
  19. 2026Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localizationcited by 0
  20. 2026Scalar Representations of Neural Network Training Dynamicscited by 0
  21. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  22. 2026The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flowcited by 0
  23. 2026Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrixcited by 0