mlsanityData to 7 Oct 2026

How many degrees of freedom do we need to train deep networks: a loss landscape perspective

arXiv 2107.05802 ↗First posted 13 Jul 2021cs.LG stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

A variety of recent works, spanning pruning, lottery tickets, and training within random subspaces, have shown that deep neural networks can be trained using far fewer degrees of freedom than the total number of parameters. We analyze this phenomenon for random subspaces by first examining the success probability of hitting a training loss sub-level set when training within a random subspace of a given training dimensionality. We find a sharp phase transition in the success probability from 0 to 1 as the training dimension surpasses a threshold. This threshold training dimension increases as the desired final loss decreases, but decreases as the initial loss decreases. We then theoretically explain the origin of this phase transition, and its dependence on initialization and final desired loss, in terms of properties of the high-dimensional geometry of the loss landscape. In particular, we show via Gordon's escape theorem, that the training dimension plus the Gaussian width of the desired loss sub-level set, projected onto a unit sphere surrounding the initialization, must exceed the total number of parameters for the success probability to be large. In several architectures and datasets, we measure the threshold training dimension as a function of initialization and demonstrate that it is a small fraction of the total parameters, implying by our theory that successful training with so few dimensions is possible precisely because the Gaussian width of low loss sub-level sets is very large. Moreover, we compare this threshold training dimension to more sophisticated ways of reducing training degrees of freedom, including lottery tickets as well as a new, analogous method: lottery subspaces. Code is available at https://github.com/ganguli-lab/degrees-of-freedom.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

26 of 38 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  8. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  9. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  10. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  11. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  12. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
14 more
  1. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  2. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  3. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  4. 2014Qualitatively characterizing neural network optimization problemscited by 186
  5. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  6. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  7. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  8. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  9. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  10. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  11. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  12. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  13. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  14. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 3; 2024: 11; 2025: 3; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  2. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 48
  3. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  4. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  5. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 37
  6. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  7. 2021Efficient Online Bayesian Inference for Neural Banditscited by 15
  8. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  9. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  10. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  11. 2024Gradient Weight-normalized Low-rank Projection for Efficient LLM Trainingcited by 11
  12. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
16 more
  1. 2023How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspectivecited by 9
  2. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8
  3. 2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
  4. 2022What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundariescited by 5
  5. 2024Identifying Policy Gradient Subspacescited by 5
  6. 2025Martingale Posterior Neural Networks for Fast Sequential Decision Makingcited by 4
  7. 2023Detecting Toxic Flowcited by 3
  8. 2023Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruningcited by 2
  9. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  10. 2025AROMA: Autonomous Rank-one Matrix Adaptationcited by 1
  11. 2024ECToNAS: Evolutionary Cross-Topology Neural Architecture Searchcited by 0
  12. 2024Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruningcited by 0
  13. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  14. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0
  15. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0
  16. 2026Apparent Compression, Real Stability: The Intrinsic Dimension of Learning a Quantum Wavefunctioncited by 0