mlsanityData to 7 Oct 2026

Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures

arXiv 2402.02697 ↗First posted 5 Feb 2024cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Deep equilibrium models (DEQs), as a typical implicit neural network, have demonstrated remarkable success on various tasks. There is, however, a lack of theoretical understanding of the connections and differences between implicit DEQs and explicit neural network models. In this paper, leveraging recent advances in random matrix theory (RMT), we perform an in-depth analysis on the eigenspectra of the conjugate kernel (CK) and neural tangent kernel (NTK) matrices for implicit DEQs, when the input data are drawn from a high-dimensional Gaussian mixture. We prove, in this setting, that the spectral behavior of these Implicit-CKs and NTKs depend on the DEQ activation function and initial weight variances, but only via a system of four nonlinear equations. As a direct consequence of this theoretical result, we demonstrate that a shallow explicit network can be carefully designed to produce the same CK or NTK as a given DEQ. Despite derived here for Gaussian mixture data, empirical results show the proposed theory and design principle also apply to popular real-world datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

35 of 57 references.

References · 35

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019Deep Equilibrium Modelscited by 603
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2021Deep learning: a statistical viewpointcited by 239
  7. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
  8. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  9. 2020Multiscale Deep Equilibrium Modelscited by 149
  10. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  11. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 126
  12. 2019Implicit Deep Learningcited by 124
23 more
  1. 2020Monotone operator equilibrium networkscited by 100
  2. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  3. 2017A Random Matrix Approach to Neural Networkscited by 80
  4. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  5. 2020The Recurrent Neural Tangent Kernelcited by 67
  6. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  7. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  8. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  9. 2023Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Modelscited by 26
  10. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  11. 2018The Dynamics of Learning: A Random Matrix Approachcited by 15
  12. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  13. 2017Invariance of Weight Distributions in Rectified MLPscited by 12
  14. 2020On Random Matrices Arising in Deep Neural Networks: General I.I.D. Casecited by 10
  15. 2022Global Convergence of Over-parameterized Deep Equilibrium Modelscited by 10
  16. 2022Deep equilibrium networks are sensitive to initialization statisticscited by 8
  17. 2021Random matrices in service of ML footprint: ternary random features with no performance losscited by 7
  18. 2023On the Neural Tangent Kernel of Equilibrium Modelscited by 7
  19. 2017A Large Dimensional Analysis of Least Squares Support Vector Machinescited by 6
  20. 2022Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscited by 5
  21. 2017A Large Dimensional Study of Regularized Discriminant Analysis Classifierscited by 4
  22. 2022Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Casecited by 4
  23. 2023Global Convergence Rate of Deep Equilibrium Models with General Activationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2025DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flowscited by 4
  2. 2024IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networkscited by 3
  3. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  4. 2024On Dissipativity of Cross-Entropy Loss in Training ResNetscited by 1
  5. 2026Consistency Deep Equilibrium Modelscited by 1
  6. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1