mlsanityData to 7 Oct 2026

Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces

arXiv 2103.11154 ↗First posted 20 Mar 2021cs.LG cs.NE math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) usually contain massive parameters, but there is redundancy such that it is guessed that the DNNs could be trained in low-dimensional subspaces. In this paper, we propose a Dynamic Linear Dimensionality Reduction (DLDR) based on low-dimensional properties of the training trajectory. The reduction is efficient, which is supported by comprehensive experiments: optimization in 40 dimensional spaces can achieve comparable performance as regular training over thousands or even millions of parameters. Since there are only a few optimization variables, we develop a quasi-Newton-based algorithm and also obtain robustness against label noises, which are two follow-up experiments to show the advantages of finding low-dimensional subspaces.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

36 of 51 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  12. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
24 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  5. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  6. 2016An overview of gradient descent optimization algorithmscited by 1,021
  7. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  11. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  12. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  13. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  14. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  15. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  16. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  17. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  18. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  19. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  20. 2018A Progressive Batching L-BFGS Method for Machine Learningcited by 45
  21. 2020Improving Neural Network Training in Low Dimensional Random Basescited by 19
  22. 2014RES: Regularized Stochastic BFGS Algorithmcited by 15
  23. 2013Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methodscited by 9
  24. 2020Quasi-Newton's method in the class gradient defined high-curvature subspacecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 0; 2024: 1; 2025: 0; 2026: 1.

5 papers.

  1. 2021Subspace Adversarial Trainingcited by 41
  2. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  3. 2024A Layer Selection Approach to Test Time Adaptationcited by 7
  4. 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimizationcited by 3
  5. 2026Distributional Spectral Diagnostics for Localizing Grokking Transitionscited by 2