mlsanityData to 14 Oct 2026

Reversible Deep Equilibrium Models

arXiv 2509.12917 ↗First posted 16 Sep 2025cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep Equilibrium Models (DEQs) are an interesting class of implicit model where the model output is implicitly defined as the fixed point of a learned function. These models have been shown to outperform explicit (fixed-depth) models in large-scale tasks by trading many deep layers for a single layer that is iterated many times. However, gradient calculation through DEQs is approximate. This often leads to unstable training dynamics and requires regularisation or many function evaluations to fix. Here, we introduce Reversible Deep Equilibrium Models (RevDEQs) that allow for exact gradient calculation, no regularisation and far fewer function evaluations than DEQs. We show that RevDEQs significantly improve performance on language modelling and image classification tasks against comparable implicit and explicit models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

40 of 52 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
28 more
  1. 2019Deep Equilibrium Modelscited by 615
  2. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  3. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  4. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  5. 2020Scalable Gradients for Stochastic Differential Equationscited by 271
  6. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  7. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
  8. 2021Neural SDEs as Infinite-Dimensional GANscited by 161
  9. 2020Multiscale Deep Equilibrium Modelscited by 151
  10. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 127
  11. 2019Implicit Deep Learningcited by 124
  12. 2020Monotone operator equilibrium networkscited by 100
  13. 2020Implicit Graph Neural Networkscited by 100
  14. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 86
  15. 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 84
  16. 2019Deep Declarative Networks: A New Hopecited by 78
  17. 2021On Training Implicit Modelscited by 75
  18. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 72
  19. 2021Efficient and Accurate Gradients for Neural SDEscited by 71
  20. 2022Deep Equilibrium Optical Flow Estimationcited by 64
  21. 2021MALI: A memory efficient and reverse accurate integrator for Neural ODEscited by 42
  22. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  23. 2021Momentum Residual Neural Networkscited by 37
  24. 2023Non-adversarial training of Neural SDEs with signature kernel scorescited by 37
  25. 2021Implicit Normalizing Flowscited by 28
  26. 2022MGNNI: Multiscale Graph Neural Networks with Implicit Layerscited by 19
  27. 2024Efficient, Accurate and Stable Gradients for Neural ODEscited by 9
  28. 2024Positive concave deep equilibrium modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 14
  2. 2025Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibriumcited by 3
  3. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  4. 2026SILVA Networks as Structured Implicit Layers and Vector Attractors via Dynamic Interaction Fieldscited by 1
  5. 2026Response Renormalization for Critical Deep Equilibrium Modelscited by 1