mlsanityData to 7 Oct 2026

Global Convergence Rate of Deep Equilibrium Models with General Activations

arXiv 2302.05797 ↗First posted 11 Feb 2023stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In a recent paper, Ling et al. investigated the over-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. This paper shows that this fact still holds for DEQs with any general activation that has bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram matrix of the equilibrium point is particularly challenging. To accomplish this task, we need to create a novel population Gram matrix and develop a new form of dual activation with Hermite polynomial expansion.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 11 Feb 2023.

Builds on

19 of 34 references.

References · 19

  1. 2019Deep Equilibrium Modelscited by 603
  2. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  5. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2020Multiscale Deep Equilibrium Modelscited by 149
  8. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  9. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 126
  10. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  11. 2018Generalization Error in Deep Learningcited by 46
  12. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
7 more
  1. 2020Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networkscited by 29
  2. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  3. 2013From average case complexity to improper learning complexitycited by 16
  4. 2022On Rademacher Complexity-based Generalization Bounds for Deep Learningcited by 16
  5. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  6. 2021Generalization Error Bounds on Deep Learning with Markov Datasetscited by 9
  7. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  2. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  3. 2023On the Equivalence between Implicit and Explicit Neural Networks: A High-dimensional Viewpointcited by 0
  4. 2026Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guaranteescited by 0