mlsanityData to 7 Oct 2026

Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology

arXiv 2002.07867 ↗First posted 18 Feb 2020cs.LG stat.MLcited by 66

Not on the map. Why?

Abstract (arXiv)

Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with N (N being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width N following the input layer suffices to ensure a similar guarantee. In particular, all the remaining layers are allowed to have constant widths, and form a pyramidal topology. We show an application of our result to the widely used LeCun's initialization and obtain an over-parameterization requirement for the single wide layer of order N2.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 18; 2023: 9; 2024: 14; 2025: 10; 2026: 3.

66 papers.

Sort:

  1. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  2. 2022Mechanistic Mode Connectivitycited by 50
  3. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  4. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  5. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  6. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  7. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  8. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  9. 2021Rethinking Influence Functions of Neural Networks in the Over-parameterized Regimecited by 26
  10. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  11. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  12. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
38 more
  1. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  2. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  3. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  4. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  5. 2023On the Convergence of Encoder-only Shallow Transformerscited by 13
  6. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  7. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  8. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  9. 2024Unraveling the Gradient Descent Dynamics of Transformerscited by 11
  10. 2021When Are Solutions Connected in Deep Networks?cited by 10
  11. 2021Local SGD Optimizes Overparameterized Neural Networks in Polynomial Timecited by 10
  12. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  13. 2022Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networkscited by 9
  14. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  15. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  16. 2022Restricted Strong Convexity of Deep Learning Models with Smooth Activationscited by 7
  17. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  18. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  19. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  20. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5
  21. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  22. 2022Finite Sample Identification of Wide Shallow Neural Networks with Biasescited by 5
  23. 2023Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networkscited by 5
  24. 2024Invertible Neural Warp for NeRFcited by 5
  25. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 5
  26. 2023On the Convergence of the Gradient Descent Method with Stochastic Fixed-point Rounding Errors under the Polyak-Lojasiewicz Inequalitycited by 4
  27. 2023On the effectiveness of neural priors in modeling dynamical systemscited by 4
  28. 2024Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activationscited by 4
  29. 2025A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Modelscited by 4
  30. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  31. 2024Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networkscited by 3
  32. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  33. 2025A Law of Data Reconstruction for Random Features (and Beyond)cited by 3
  34. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  35. 2021Generalization of GANs and overparameterized models under Lipschitz continuitycited by 2
  36. 2022Finite-Sum Optimization: A New Perspective for Convergence to a Global Solutioncited by 2
  37. 2022A Framework for Overparameterized Learningcited by 2
  38. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2

50 most cited of 66 All 66 (JSON)