mlsanityData to 7 Oct 2026

Global Convergence of Over-parameterized Deep Equilibrium Models

arXiv 2205.13814 ↗First posted 27 May 2022cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

A deep equilibrium model (DEQ) is implicitly defined through an equilibrium point of an infinite-depth weight-tied model with an input-injection. Instead of infinite computations, it solves an equilibrium point directly with root-finding and computes gradients with implicit differentiation. The training dynamics of over-parameterized DEQs are investigated in this study. By supposing a condition on the initial equilibrium point, we show that the unique equilibrium point always exists during the training process, and the gradient descent is proved to converge to a globally optimal solution at a linear convergence rate for the quadratic loss function. In order to show that the required initial condition is satisfied via mild over-parameterization, we perform a fine-grained analysis on random DEQs. We propose a novel probabilistic framework to overcome the technical difficulty in the non-asymptotic analysis of infinite-depth weight-tied models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

25 of 34 references.

References · 25

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2019Deep Equilibrium Modelscited by 603
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  8. 2020Multiscale Deep Equilibrium Modelscited by 149
  9. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  11. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  12. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
13 more
  1. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 126
  2. 2019Implicit Deep Learningcited by 124
  3. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  4. 2020Monotone operator equilibrium networkscited by 100
  5. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  6. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  7. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  8. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  9. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  10. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  11. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  12. 2021On the Provable Generalization of Recurrent Neural Networkscited by 6
  13. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 0; 2026: 3.

10 papers.

  1. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  2. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  3. 2023Global Convergence Rate of Deep Equilibrium Models with General Activationscited by 4
  4. 2024IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networkscited by 3
  5. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  6. 2024Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothingcited by 2
  7. 2026Consistency Deep Equilibrium Modelscited by 1
  8. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  9. 2023On the Equivalence between Implicit and Explicit Neural Networks: A High-dimensional Viewpointcited by 0
  10. 2026Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guaranteescited by 0