# Monotone operator equilibrium networks

arXiv: https://arxiv.org/abs/2006.08591 · HTML: https://mlsanity.com/paper/2006.08591/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 30 of 100)

- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modeling](https://mlsanity.com/paper/2203.08205/) (2022)
- [Deep Equilibrium Optical Flow Estimation](https://mlsanity.com/paper/2204.08442/) (2022)
- [One-Step Diffusion Distillation via Deep Equilibrium Models](https://mlsanity.com/paper/2401.08639/) (2023)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network](https://mlsanity.com/paper/2201.02217/) (2022)
- [Nonsmooth Implicit Differentiation for Machine Learning and Optimization](https://mlsanity.com/paper/2106.04350/) (2021)
- [Deep Equilibrium Approaches to Diffusion Models](https://mlsanity.com/paper/2210.12867/) (2022)
- [Path Independent Equilibrium Models Can Better Exploit Test-Time Computation](https://mlsanity.com/paper/2211.09961/) (2022)
- [Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models](https://mlsanity.com/paper/2304.00600/) (2023)
- [Online Deep Equilibrium Learning for Regularization by Denoising](https://mlsanity.com/paper/2205.13051/) (2022)
- [TorchDEQ: A Library for Deep Equilibrium Models](https://mlsanity.com/paper/2310.18605/) (2023)
- [Robust Implicit Networks via Non-Euclidean Contractions](https://mlsanity.com/paper/2106.03194/) (2021)
- [Deep Equilibrium Based Neural Operators for Steady-State PDEs](https://mlsanity.com/paper/2312.00234/) (2023)
- [Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers](https://mlsanity.com/paper/2508.10480/) (2025)
- [Feasibility-based Fixed Point Networks](https://mlsanity.com/paper/2104.14090/) (2021)
- [Joint inference and input optimization in equilibrium networks](https://mlsanity.com/paper/2111.13236/) (2021)
- [A Closer Look at the Adversarial Robustness of Deep Equilibrium Models](https://mlsanity.com/paper/2306.01429/) (2023)
- [Convergent Graph Solvers](https://mlsanity.com/paper/2106.01680/) (2021)
- [A nonlinear diffusion method for semi-supervised learning on hypergraphs](https://mlsanity.com/paper/2103.14867/) (2021)
- [Global Convergence of Over-parameterized Deep Equilibrium Models](https://mlsanity.com/paper/2205.13814/) (2022)
- [Lyapunov-Stable Deep Equilibrium Models](https://mlsanity.com/paper/2304.12707/) (2023)
- [Differentiation Through Black-Box Quadratic Programming Solvers](https://mlsanity.com/paper/2410.06324/) (2024)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach](https://mlsanity.com/paper/2112.05310/) (2021)
- [Deep equilibrium networks are sensitive to initialization statistics](https://mlsanity.com/paper/2207.09432/) (2022)

## Abstract (arXiv)

Implicit-depth models such as Deep Equilibrium Networks have recently been shown to match or exceed the performance of traditional deep networks while being much more memory efficient. However, these models suffer from unstable convergence to a solution and lack guarantees that a solution exists. On the other hand, Neural ODEs, another class of implicit-depth models, do guarantee existence of a unique solution but perform poorly compared with traditional networks. In this paper, we develop a new class of implicit-depth model based on the theory of monotone operators, the Monotone Operator Equilibrium Network (monDEQ). We show the close connection between finding the equilibrium point of an implicit network and solving a form of monotone operator splitting problem, which admits efficient solvers with guaranteed, stable convergence. We then develop a parameterization of the network which ensures that all operators remain monotone, which guarantees the existence of a unique equilibrium point. Finally, we show how to instantiate several versions of these models, and implement the resulting iterative solvers, for structured linear operators such as multi-scale convolutions. The resulting models vastly outperform the Neural ODE-based models while also being more computationally efficient. Code is available at http://github.com/locuslab/monotone_op_net.