mlsanityData to 7 Oct 2026

Deep Equilibrium Models

arXiv 1909.01377 ↗First posted 3 Sep 2019cs.LG stat.MLcited by 603

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is equivalent to running an infinite depth (weight-tied) feedforward network, but has the notable advantage that we can analytically backpropagate through the equilibrium point using implicit differentiation. Using this approach, training and prediction in these networks require only constant memory, regardless of the effective "depth" of the network. We demonstrate how DEQs can be applied to two state-of-the-art deep sequence models: self-attention transformers and trellis networks. On large-scale language modeling tasks, such as the WikiText-103 benchmark, we show that DEQs 1) often improve performance over these state-of-the-art models (for similar parameter counts); 2) have similar computational requirements to existing models; and 3) vastly reduce memory consumption (often the bottleneck for training large sequence models), demonstrating an up-to 88% memory reduction in our experiments. The code is available at https://github.com/locuslab/deq .

Key facts. Cited by 603 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 99; 2022: 104; 2023: 78; 2024: 87; 2025: 75; 2026: 160.

603 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2022Transformers learn in-context by gradient descentcited by 714
  3. 2021Implicit Behavioral Cloningcited by 442
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  5. 2021Advances in Neural Renderingcited by 365
  6. 2021GFlowNet Foundationscited by 286
  7. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  8. 2021Efficient and Modular Implicit Differentiationcited by 199
  9. 2021DC3: A learning method for optimization with hard constraintscited by 181
  10. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  11. 2025Hierarchical Reasoning Modelcited by 167
  12. 2021Pretrained Transformers as Universal Computation Enginescited by 162
38 more
  1. 2021Neural SDEs as Infinite-Dimensional GANscited by 159
  2. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  3. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  4. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  5. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 126
  6. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  7. 2021Neural Flows: Efficient Alternative to Neural ODEscited by 99
  8. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  9. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  10. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  11. 2021You Only Learn One Representation: Unified Network for Multiple Taskscited by 86
  12. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  13. 2022Tutorial on amortized optimizationcited by 83
  14. 2021Gradients are Not All You Needcited by 82
  15. 2023A Survey of Contextual Optimization Methods for Decision Making under Uncertaintycited by 81
  16. 2021Stiff Neural Ordinary Differential Equationscited by 77
  17. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  18. 2021On Training Implicit Modelscited by 73
  19. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  20. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  21. 2023Energy Transformercited by 69
  22. 2022Neural Conservation Laws: A Divergence-Free Perspectivecited by 67
  23. 2021Closed-form Continuous-time Neural Modelscited by 65
  24. 2022Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modelingcited by 65
  25. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  26. 2022Deep Equilibrium Optical Flow Estimationcited by 63
  27. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  28. 2023One-Step Diffusion Distillation via Deep Equilibrium Modelscited by 60
  29. 2021Smooth Normalizing Flowscited by 52
  30. 2021A Survey on Green Deep Learningcited by 52
  31. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  32. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  33. 2022Rank Diminishing in Deep Neural Networkscited by 47
  34. 2021Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamicscited by 46
  35. 2022Learned reconstruction methods with convergence guaranteescited by 46
  36. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  37. 2022Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Networkcited by 45
  38. 2022Equivariant Hypergraph Diffusion Neural Operatorscited by 45

50 most cited of 603 All 603 (JSON)