mlsanityData to 14 Oct 2026

Gradient Descent Happens in a Tiny Subspace

arXiv 1812.04754 ↗First posted 12 Dec 2018cs.LG cs.AI stat.MLcited by 217

Not on the map. Why?

Abstract (arXiv)

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 29; 2022: 29; 2023: 23; 2024: 41; 2025: 25; 2026: 69.

217 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  3. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  4. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  7. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  8. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  9. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  10. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  11. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  12. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
38 more
  1. 2022Generalizing to New Physical Systems via Context-Informed Dynamics Modelcited by 50
  2. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  3. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  4. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  5. 2021A study on the plasticity of neural networkscited by 44
  6. 2021Subspace Adversarial Trainingcited by 41
  7. 2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
  8. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  9. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  10. 2024Does SGD really happen in tiny subspaces?cited by 36
  11. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  12. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  13. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  14. 2021Power-law escape rate of SGDcited by 29
  15. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 28
  16. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  17. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 26
  18. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  19. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  20. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  21. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  22. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  23. 2024Machine Unlearning via Null Space Calibrationcited by 21
  24. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  25. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  26. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  27. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  28. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  29. 2021An Operator Theoretic View on Pruning Deep Neural Networkscited by 18
  30. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  31. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 17
  32. 2021Privately Learning Subspacescited by 16
  33. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  34. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  35. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 16
  36. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  37. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  38. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16

50 most cited of 217 All 217 (JSON)