mlsanityData to 7 Oct 2026

On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

arXiv 1802.06509 ↗First posted 19 Feb 2018cs.LGcited by 280

Not on the map. Why?

Abstract (arXiv)

Conventional wisdom in deep learning states that increasing depth improves expressiveness but complicates optimization. This paper suggests that, sometimes, increasing depth can speed up optimization. The effect of depth on optimization is decoupled from expressiveness by focusing on settings where additional layers amount to overparameterization - linear neural networks, a well-studied model. Theoretical analysis, as well as experiments, show that here depth acts as a preconditioner which may accelerate convergence. Even on simple convex problems such as linear regression with ℓp loss, p>2, gradient descent can benefit from transitioning to a non-convex overparameterized objective, more than it would from some common acceleration schemes. We also prove that it is mathematically impossible to obtain the acceleration effect of overparametrization via gradients of any regularizer.

Key facts. Cited by 280 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 66; 2022: 53; 2023: 48; 2024: 39; 2025: 33; 2026: 40.

280 papers.

Sort:

By area:

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  4. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  5. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  6. 2021Towards Understanding Knowledge Distillationcited by 158
  7. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  8. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
  9. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  10. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  11. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  12. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
38 more
  1. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  2. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  3. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  4. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  5. 2022q-Learning in Continuous Timecited by 68
  6. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  7. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  8. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  9. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  10. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  11. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  12. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  13. 2023Transformers learn through gradual rank increasecited by 52
  14. 2022Mechanistic Mode Connectivitycited by 50
  15. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  16. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  17. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  18. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  19. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  20. 2021Implicit Regularization in Tensor Factorizationcited by 43
  21. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  22. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  23. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  24. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  25. 2023A Survey on Efficient Training of Transformerscited by 37
  26. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  27. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  28. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  29. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  30. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  31. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  32. 2021KNAS: Green Neural Architecture Searchcited by 33
  33. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  34. 2025Symmetry in Neural Network Parameter Spacescited by 31
  35. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  36. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  37. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  38. 2021Geometry of Linear Convolutional Networkscited by 27

50 most cited of 280 All 280 (JSON)