# Lipschitz Bounded Equilibrium Networks

arXiv: https://arxiv.org/abs/2010.01732 · HTML: https://mlsanity.com/paper/2010.01732/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 30 of 49)

- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [One-Step Diffusion Distillation via Deep Equilibrium Models](https://mlsanity.com/paper/2401.08639/) (2023)
- [A Unified Algebraic Perspective on Lipschitz Neural Networks](https://mlsanity.com/paper/2303.03169/) (2023)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [Path Independent Equilibrium Models Can Better Exploit Test-Time Computation](https://mlsanity.com/paper/2211.09961/) (2022)
- [Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Models](https://mlsanity.com/paper/2304.00600/) (2023)
- [Some Fundamental Aspects about Lipschitz Continuity of Neural Networks](https://mlsanity.com/paper/2302.10886/) (2023)
- [TorchDEQ: A Library for Deep Equilibrium Models](https://mlsanity.com/paper/2310.18605/) (2023)
- [Robust Implicit Networks via Non-Euclidean Contractions](https://mlsanity.com/paper/2106.03194/) (2021)
- [Deep Equilibrium Based Neural Operators for Steady-State PDEs](https://mlsanity.com/paper/2312.00234/) (2023)
- [Joint inference and input optimization in equilibrium networks](https://mlsanity.com/paper/2111.13236/) (2021)
- [A Closer Look at the Adversarial Robustness of Deep Equilibrium Models](https://mlsanity.com/paper/2306.01429/) (2023)
- [Global Convergence of Over-parameterized Deep Equilibrium Models](https://mlsanity.com/paper/2205.13814/) (2022)
- [Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramian](https://mlsanity.com/paper/2303.11835/) (2023)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach](https://mlsanity.com/paper/2112.05310/) (2021)
- [Deep equilibrium networks are sensitive to initialization statistics](https://mlsanity.com/paper/2207.09432/) (2022)
- [Lipschitz constant estimation for general neural network architectures using control tools](https://mlsanity.com/paper/2405.01125/) (2024)
- [RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks](https://mlsanity.com/paper/2106.08928/) (2021)
- [Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations](https://mlsanity.com/paper/2401.14033/) (2024)
- [Positive concave deep equilibrium models](https://mlsanity.com/paper/2402.04029/) (2024)
- [Optimization Induced Equilibrium Networks](https://mlsanity.com/paper/2105.13228/) (2021)
- [Convolutional Neural Networks as 2-D systems](https://mlsanity.com/paper/2303.03042/) (2023)
- [Deep equilibrium models as estimators for continuous latent variables](https://mlsanity.com/paper/2211.05943/) (2022)
- [On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks](https://mlsanity.com/paper/2405.11432/) (2024)
- [State space representations of the Roesser type for convolutional layers](https://mlsanity.com/paper/2403.11938/) (2024)
- [Quantization Robustness of Monotone Operator Equilibrium Networks](https://mlsanity.com/paper/2603.10562/) (2026)
- [Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks](https://mlsanity.com/paper/2204.00187/) (2022)

## Abstract (arXiv)

This paper introduces new parameterizations of equilibrium neural networks, i.e. networks defined by implicit equations. This model class includes standard multilayer and residual networks as special cases. The new parameterization admits a Lipschitz bound during training via unconstrained optimization: no projections or barrier functions are required. Lipschitz bounds are a common proxy for robustness and appear in many generalization bounds. Furthermore, compared to previous works we show well-posedness (existence of solutions) under less restrictive conditions on the network weights and more natural assumptions on the activation functions: that they are monotone and slope restricted. These results are proved by establishing novel connections with convex optimization, operator splitting on non-Euclidean spaces, and contracting neural ODEs. In image classification experiments we show that the Lipschitz bounds are very accurate and improve robustness to adversarial attacks.