mlsanityData to 7 Oct 2026

On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers

arXiv 2102.07346 ↗First posted 15 Feb 2021cs.LG cs.AI cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

A deep equilibrium model uses implicit layers, which are implicitly defined through an equilibrium point of an infinite sequence of computation. It avoids any explicit computation of the infinite sequence by finding an equilibrium point directly via root-finding and by computing gradients via implicit differentiation. In this paper, we analyze the gradient dynamics of deep equilibrium models with nonlinearity only on weight matrices and non-convex objective functions of weights for regression and classification. Despite non-convexity, convergence to global optimum at a linear rate is guaranteed without any assumption on the width of the models, allowing the width to be smaller than the output dimension and the number of data points. Moreover, we prove a relation between the gradient dynamics of the deep implicit layer and the dynamics of trust region Newton method of a shallow explicit layer. This mathematically proven relation along with our numerical observation suggests the importance of understanding implicit bias of implicit layers and an open problem on the topic. Our proofs deal with implicit layers, weight tying and nonlinearity on weights, and differ from those in the related literature.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

46 of 63 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  8. 2018Universal Transformerscited by 613
  9. 2019Deep Equilibrium Modelscited by 603
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
34 more
  1. 2018Deep Learning for Classical Japanese Literaturecited by 415
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2017Implicit Regularization in Matrix Factorizationcited by 318
  4. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  5. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  8. 2016Deep Learning without Poor Local Minimacited by 233
  9. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  10. 2020Directional convergence and alignment in deep learningcited by 172
  11. 2017Generalization in Deep Learningcited by 146
  12. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  13. 2016Identity Matters in Deep Learningcited by 107
  14. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  15. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  16. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  17. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  18. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  19. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  20. 2018Trellis Networks for Sequence Modelingcited by 50
  21. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  22. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  23. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  24. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  25. 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
  26. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  27. 2019GraphMix: Improved Training of GNNs for Semi-Supervised Learningcited by 23
  28. 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
  29. 2018Depth with Nonlinearity Creates No Bad Local Minima in ResNetscited by 17
  30. 2018Are ResNets Provably Better than Linear Predictors?cited by 12
  31. 2018Effect of Depth and Width on Local Minima in Deep Learningcited by 12
  32. 2019Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networkscited by 11
  33. 2019Elimination of All Bad Local Minima in Deep Learningcited by 10
  34. 2021A Note on Connectivity of Sublevel Sets in Deep Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 14; 2023: 4; 2024: 3; 2025: 2; 2026: 4.

35 papers.

Sort:

By area:

  1. 2021On Training Implicit Modelscited by 73
  2. 2022Deep Equilibrium Optical Flow Estimationcited by 63
  3. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 57
  4. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  5. 2022Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodologycited by 43
  6. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  7. 2022EIGNN: Efficient Infinite-Depth Graph Neural Networkscited by 32
  8. 2022Online Deep Equilibrium Learning for Regularization by Denoisingcited by 22
  9. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  10. 2021Robust Implicit Networks via Non-Euclidean Contractionscited by 19
  11. 2022MGNNI: Multiscale Graph Neural Networks with Implicit Layerscited by 19
  12. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
23 more
  1. 2021Joint inference and input optimization in equilibrium networkscited by 15
  2. 2021Understanding Dynamics of Nonlinear Representation Learning and Its Applicationcited by 11
  3. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  4. 2024Differentiation Through Black-Box Quadratic Programming Solverscited by 10
  5. 2022Self-Distillation for Further Pre-training of Transformerscited by 9
  6. 2021A Recipe for Global Convergence Guarantee in Deep Neural Networkscited by 8
  7. 2022Deep equilibrium networks are sensitive to initialization statisticscited by 8
  8. 2022Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinitycited by 5
  9. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5
  10. 2022Stable Invariant Models via Koopman Spectracited by 4
  11. 2023iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixercited by 3
  12. 2023Test like you Train in Implicit Deep Learningcited by 3
  13. 2025IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Methodcited by 3
  14. 2022On the optimization and generalization of overparameterized implicit neural networkscited by 2
  15. 2025Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lenscited by 2
  16. 2024Score-fPINN: Fractional Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck-Levy Equationscited by 1
  17. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  18. 2026Implicit Machine Learning Force Fields Accelerate Molecular Dynamics Simulationscited by 1
  19. 2022Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensionscited by 0
  20. 2023Federated Deep Equilibrium Learning: Harnessing Compact Global Representations to Enhance Personalizationcited by 0
  21. 2024Training Implicit Networks for Image Deblurring using Jacobian-Free Backpropagationcited by 0
  22. 2026Eigenanalysis framework for autoregressive neural emulators of multi-scale chaotic dynamicscited by 0
  23. 2026Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guaranteescited by 0