mlsanityData to 14 Oct 2026

TorchDEQ: A Library for Deep Equilibrium Models

arXiv 2310.18605 ↗First posted 28 Oct 2023cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Deep Equilibrium (DEQ) Models, an emerging class of implicit models that maps inputs to fixed points of neural networks, are of growing interest in the deep learning community. However, training and applying DEQ models is currently done in an ad-hoc fashion, with various techniques spread across the literature. In this work, we systematically revisit DEQs and present TorchDEQ, an out-of-the-box PyTorch-based library that allows users to define, train, and infer using DEQs over multiple domains with minimal code and best practices. Using TorchDEQ, we build a ``DEQ Zoo'' that supports six published implicit models across different domains. By developing a joint framework that incorporates the best practices across all models, we have substantially improved the performance, training stability, and efficiency of DEQs on ten datasets across all six projects in the DEQ Zoo. TorchDEQ and DEQ Zoo are released as hrefhttps://github.com/locuslab/torchdeqopen source.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 84 of its 107 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

84 of 107 references.

References · 84

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2017Graph Attention Networkscited by 8,923
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2017Inductive Representation Learning on Large Graphscited by 6,320
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  7. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  12. 2018Group Normalizationcited by 1,524
  13. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  14. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  15. 2020On Layer Normalization in the Transformer Architecturecited by 922
  16. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  17. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  18. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  19. 2019Deep Equilibrium Modelscited by 615
  20. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  21. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  22. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  23. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  24. 2021Random Feature Attentioncited by 354
  25. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  26. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 327
  27. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 275
  28. 2022Unifying Flow, Stereo and Depth Estimationcited by 275
  29. 2021Learning to Estimate Hidden Motions with Global Motion Aggregationcited by 224
  30. 2021Efficient and Modular Implicit Differentiationcited by 203
  31. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  32. 2020Multiscale Deep Equilibrium Modelscited by 151
  33. 2022MetaFormer Baselines for Visioncited by 132
  34. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 127
  35. 2019Implicit Deep Learningcited by 124
  36. 2020Monotone operator equilibrium networkscited by 100
  37. 2020Implicit Graph Neural Networkscited by 100
  38. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 6; 2026: 6.

21 papers.

Sort:

By area:

  1. 2025Less is More: Recursive Reasoning with Tiny Networkscited by 45
  2. 2024Fixed Point Diffusion Modelscited by 16
  3. 2024Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Modelscited by 6
  4. 2024Deep Equilibrium Algorithmic Reasoningcited by 6
  5. 2024Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registrationcited by 5
  6. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4
  7. 2025DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flowscited by 4
  8. 2024Learning of Patch-Based Smooth-Plus-Sparse Models for Image Reconstructioncited by 3
  9. 2025DEQuify your force field: More efficient simulations using deep equilibrium modelscited by 3
  10. 2024Quantum Deep Equilibrium Modelscited by 2
  11. 2024Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothingcited by 2
  12. 2025Bridging Classical and Learning-based Iterative Registration through Deep Equilibrium Modelscited by 2
9 more
  1. 2025Multivariate Fields of Experts for Convergent Image Reconstructioncited by 2
  2. 2024The Deep Equilibrium Algorithmic Reasonercited by 1
  3. 2025Equivariant Deep Equilibrium Models for Imaging Inverse Problemscited by 1
  4. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  5. 2026Response Renormalization for Critical Deep Equilibrium Modelscited by 1
  6. 2026Learned Finite Element-based Regularization of the Inverse Problem in Electrocardiographic Imagingcited by 0
  7. 2026Vision Tiny Recursion Model (ViTRM): Parameter-Efficient Image Classification via Recursive State Refinementcited by 0
  8. 2026Visual Implicit Autoregressive Modelingcited by 0
  9. 2026Updating the standard neuron model in artificial neural networkscited by 0