# Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures

arXiv: https://arxiv.org/abs/2402.02697 · HTML: https://mlsanity.com/paper/2402.02697/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 57 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 30 of 35)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels](https://mlsanity.com/paper/1905.13192/) (2019)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks](https://mlsanity.com/paper/2005.11879/) (2020)
- [The Recurrent Neural Tangent Kernel](https://mlsanity.com/paper/2006.10246/) (2020)
- [Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures](https://mlsanity.com/paper/2001.08370/) (2020)
- [On the Spectrum of Random Features Maps of High Dimensional Data](https://mlsanity.com/paper/1805.11916/) (2018)
- [Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models](https://mlsanity.com/paper/2304.00600/) (2023)
- [SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models](https://mlsanity.com/paper/2106.00553/) (2021)
- [A global convergence theory for deep ReLU implicit networks via over-parameterization](https://mlsanity.com/paper/2110.05645/) (2021)
- [The Dynamics of Learning: A Random Matrix Approach](https://mlsanity.com/paper/1805.11917/) (2018)
- [Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models](https://mlsanity.com/paper/2310.10767/) (2023)
- [Invariance of Weight Distributions in Rectified MLPs](https://mlsanity.com/paper/1711.09090/) (2017)
- [On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case](https://mlsanity.com/paper/2011.11439/) (2020)
- [Global Convergence of Over-parameterized Deep Equilibrium Models](https://mlsanity.com/paper/2205.13814/) (2022)
- [Deep equilibrium networks are sensitive to initialization statistics](https://mlsanity.com/paper/2207.09432/) (2022)
- [A Large Dimensional Analysis of Least Squares Support Vector Machines](https://mlsanity.com/paper/1701.02967/) (2017)
- [Random matrices in service of ML footprint: ternary random features with no performance loss](https://mlsanity.com/paper/2110.01899/) (2021)

## Built on by (most cited first, 6 of 6)

- [DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flows](https://mlsanity.com/paper/2503.01140/) (2025)
- [IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networks](https://mlsanity.com/paper/2410.08524/) (2024)
- [Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective](https://mlsanity.com/paper/2410.23391/) (2024)
- [Consistency Deep Equilibrium Models](https://mlsanity.com/paper/2602.03024/) (2026)
- [On Dissipativity of Cross-Entropy Loss in Training ResNets](https://mlsanity.com/paper/2405.19013/) (2024)
- [Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach](https://mlsanity.com/paper/2602.03297/) (2026)

## Abstract (arXiv)

Deep equilibrium models (DEQs), as a typical implicit neural network, have demonstrated remarkable success on various tasks. There is, however, a lack of theoretical understanding of the connections and differences between implicit DEQs and explicit neural network models. In this paper, leveraging recent advances in random matrix theory (RMT), we perform an in-depth analysis on the eigenspectra of the conjugate kernel (CK) and neural tangent kernel (NTK) matrices for implicit DEQs, when the input data are drawn from a high-dimensional Gaussian mixture. We prove, in this setting, that the spectral behavior of these Implicit-CKs and NTKs depend on the DEQ activation function and initial weight variances, but only via a system of four nonlinear equations. As a direct consequence of this theoretical result, we demonstrate that a shallow explicit network can be carefully designed to produce the same CK or NTK as a given DEQ. Despite derived here for Gaussian mixture data, empirical results show the proposed theory and design principle also apply to popular real-world datasets.