mlsanityData to 7 Oct 2026

Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widths

arXiv 2205.07463 ↗First posted 16 May 2022cs.LG math.OC stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Implicit deep learning has recently become popular in the machine learning community since these implicit models can achieve competitive performance with state-of-the-art deep networks while using significantly less memory and computational resources. However, our theoretical understanding of when and how first-order methods such as gradient descent (GD) converge on textitnonlinear implicit networks is limited. Although this type of problem has been studied in standard feed-forward networks, the case of implicit models is still intriguing because implicit networks have textitinfinitely many layers. The corresponding equilibrium equation probably admits no or multiple solutions during training. This paper studies the convergence of both gradient flow (GF) and gradient descent for nonlinear ReLU activated implicit networks. To deal with the well-posedness problem, we introduce a fixed scalar to scale the weight matrix of the implicit layer and show that there exists a small enough scaling constant, keeping the equilibrium equation well-posed throughout training. As a result, we prove that both GF and GD converge to a global minimum at a linear rate if the width m of the implicit network is textitlinear in the sample size N, i.e., m=Omega(N).

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 16 May 2022.

Builds on

21 of 27 references.

References · 21

  1. 2018Neural Ordinary Differential Equationscited by 3,733
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2019Deep Equilibrium Modelscited by 603
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2020Multiscale Deep Equilibrium Modelscited by 149
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  11. 2019Implicit Deep Learningcited by 124
  12. 2020Monotone operator equilibrium networkscited by 100
9 more
  1. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  2. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  3. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  4. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  5. 2018Trellis Networks for Sequence Modelingcited by 50
  6. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  7. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  8. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  9. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 1.

5 papers.

  1. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  2. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  3. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  4. 2023Global Convergence Rate of Deep Equilibrium Models with General Activationscited by 4
  5. 2022On the optimization and generalization of overparameterized implicit neural networkscited by 2