# TorchDEQ: A Library for Deep Equilibrium Models

arXiv: https://arxiv.org/abs/2310.18605 · HTML: https://mlsanity.com/paper/2310.18605/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 107 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 30 of 84)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)

## Built on by (most cited first, 21 of 21)

- [Less is More: Recursive Reasoning with Tiny Networks](https://mlsanity.com/paper/2510.04871/) (2025)
- [Fixed Point Diffusion Models](https://mlsanity.com/paper/2401.08741/) (2024)
- [Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models](https://mlsanity.com/paper/2406.03794/) (2024)
- [Deep Equilibrium Algorithmic Reasoning](https://mlsanity.com/paper/2410.15059/) (2024)
- [Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration](https://mlsanity.com/paper/2406.07361/) (2024)
- [Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling](https://mlsanity.com/paper/2410.01440/) (2024)
- [DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flows](https://mlsanity.com/paper/2503.01140/) (2025)
- [Learning of Patch-Based Smooth-Plus-Sparse Models for Image Reconstruction](https://mlsanity.com/paper/2412.13070/) (2024)
- [DEQuify your force field: More efficient simulations using deep equilibrium models](https://mlsanity.com/paper/2509.08734/) (2025)
- [Quantum Deep Equilibrium Models](https://mlsanity.com/paper/2410.23940/) (2024)
- [Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing](https://mlsanity.com/paper/2411.00899/) (2024)
- [Bridging Classical and Learning-based Iterative Registration through Deep Equilibrium Models](https://mlsanity.com/paper/2507.00582/) (2025)
- [Multivariate Fields of Experts for Convergent Image Reconstruction](https://mlsanity.com/paper/2508.06490/) (2025)
- [The Deep Equilibrium Algorithmic Reasoner](https://mlsanity.com/paper/2402.06445/) (2024)
- [Equivariant Deep Equilibrium Models for Imaging Inverse Problems](https://mlsanity.com/paper/2511.18667/) (2025)
- [Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach](https://mlsanity.com/paper/2602.03297/) (2026)
- [Response Renormalization for Critical Deep Equilibrium Models](https://mlsanity.com/paper/2608.23725/) (2026)
- [Learned Finite Element-based Regularization of the Inverse Problem in Electrocardiographic Imaging](https://mlsanity.com/paper/2602.07466/) (2026)
- [Vision Tiny Recursion Model (ViTRM): Parameter-Efficient Image Classification via Recursive State Refinement](https://mlsanity.com/paper/2603.19503/) (2026)
- [Visual Implicit Autoregressive Modeling](https://mlsanity.com/paper/2605.01220/) (2026)
- [Updating the standard neuron model in artificial neural networks](https://mlsanity.com/paper/2605.30370/) (2026)

## Abstract (arXiv)

Deep Equilibrium (DEQ) Models, an emerging class of implicit models that maps inputs to fixed points of neural networks, are of growing interest in the deep learning community. However, training and applying DEQ models is currently done in an ad-hoc fashion, with various techniques spread across the literature. In this work, we systematically revisit DEQs and present TorchDEQ, an out-of-the-box PyTorch-based library that allows users to define, train, and infer using DEQs over multiple domains with minimal code and best practices. Using TorchDEQ, we build a ``DEQ Zoo'' that supports six published implicit models across different domains. By developing a joint framework that incorporates the best practices across all models, we have substantially improved the performance, training stability, and efficiency of DEQs on ten datasets across all six projects in the DEQ Zoo. TorchDEQ and DEQ Zoo are released as hrefhttps://github.com/locuslab/torchdeqopen source.