# Deep equilibrium models as estimators for continuous latent variables

arXiv: https://arxiv.org/abs/2211.05943 · HTML: https://mlsanity.com/paper/2211.05943/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 55 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Deep Declarative Networks: A New Hope](https://mlsanity.com/paper/1909.04866/) (2019)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Lipschitz Bounded Equilibrium Networks](https://mlsanity.com/paper/2010.01732/) (2020)
- [Deep Equilibrium Approaches to Diffusion Models](https://mlsanity.com/paper/2210.12867/) (2022)
- [The continuous Bernoulli: fixing a pervasive error in variational autoencoders](https://mlsanity.com/paper/1907.06845/) (2019)
- [Periodic Activation Functions Induce Stationarity](https://mlsanity.com/paper/2110.13572/) (2021)
- [Regularization of Inverse Problems: Deep Equilibrium Models versus Bilevel Learning](https://mlsanity.com/paper/2206.13193/) (2022)
- [Loop Unrolled Shallow Equilibrium Regularizer (LUSER) -- A Memory-Efficient Inverse Problem Solver](https://mlsanity.com/paper/2210.04987/) (2022)

## Built on by (most cited first, 4 of 4)

- [One-Step Diffusion Distillation via Deep Equilibrium Models](https://mlsanity.com/paper/2401.08639/) (2023)
- [Lyapunov-Stable Deep Equilibrium Models](https://mlsanity.com/paper/2304.12707/) (2023)
- [Positive concave deep equilibrium models](https://mlsanity.com/paper/2402.04029/) (2024)
- [Monotone deep Boltzmann machines](https://mlsanity.com/paper/2307.04990/) (2023)

## Abstract (arXiv)

Principal Component Analysis (PCA) and its exponential family extensions have three components: observations, latents and parameters of a linear transformation. We consider a generalised setting where the canonical parameters of the exponential family are a nonlinear transformation of the latents. We show explicit relationships between particular neural network architectures and the corresponding statistical models. We find that deep equilibrium models -- a recently introduced class of implicit neural networks -- solve maximum a-posteriori (MAP) estimates for the latents and parameters of the transformation. Our analysis provides a systematic way to relate activation functions, dropout, and layer structure, to statistical assumptions about the observations, thus providing foundational principles for unsupervised DEQs. For hierarchical latents, individual neurons can be interpreted as nodes in a deep graphical model. Our DEQ feature maps are end-to-end differentiable, enabling fine-tuning for downstream tasks.