mlsanityData to 14 Oct 2026

On Training Implicit Models

arXiv 2111.05177 ↗First posted 9 Nov 2021cs.LGcited by 75

Not on the map. Why?

Abstract (arXiv)

This paper focuses on training implicit models of infinite layers. Specifically, previous works employ implicit differentiation and solve the exact gradient for the backward propagation. However, is it necessary to compute such an exact but expensive gradient for training? In this work, we propose a novel gradient estimate for implicit models, named phantom gradient, that 1) forgoes the costly computation of the exact gradient; and 2) provides an update direction empirically preferable to the implicit model training. We theoretically analyze the condition under which an ascent direction of the loss landscape could be found, and provide two specific instantiations of the phantom gradient based on the damped unrolling and Neumann series. Experiments on large-scale tasks demonstrate that these lightweight phantom gradients significantly accelerate the backward passes in training implicit models by roughly 1.7 times, and even boost the performance over approaches based on the exact gradient on ImageNet.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

46 of 65 references.

References · 46

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  8. 2019Deep Equilibrium Modelscited by 615
  9. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  10. 2019Meta-Learning with Implicit Gradientscited by 525
  11. 2019Augmented Neural ODEscited by 397
  12. 2018Invertible Residual Networkscited by 365
34 more
  1. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  2. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  3. 2019Residual Flows for Invertible Generative Modelingcited by 257
  4. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  5. 2016Hyperparameter optimization with approximate gradientcited by 233
  6. 2019Differentiation of Blackbox Combinatorial Solverscited by 226
  7. 2018Truncated Back-propagation for Bilevel Optimizationcited by 190
  8. 2019SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solvercited by 173
  9. 2020Multiscale Deep Equilibrium Modelscited by 151
  10. 2014Difference Target Propagationcited by 148
  11. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
  12. 2019Implicit Deep Learningcited by 124
  13. 2020Monotone operator equilibrium networkscited by 100
  14. 2020Implicit Graph Neural Networkscited by 100
  15. 2017Shake-Shake regularizationcited by 87
  16. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 79
  17. 2019Deep Declarative Networks: A New Hopecited by 78
  18. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  19. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 72
  20. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  21. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  22. 2020A Theoretical Framework for Target Propagationcited by 57
  23. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  24. 2019End to end learning and optimization on graphscited by 44
  25. 2017Understanding Synthetic Gradients and Decoupled Neural Interfacescited by 35
  26. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  27. 2021Implicit Normalizing Flowscited by 28
  28. 2019Learning to solve the credit assignment problemcited by 27
  29. 2017Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networkscited by 26
  30. 2018Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Trainingcited by 19
  31. 2020Implicit Feature Pyramid Network for Object Detectioncited by 15
  32. 2018Lifted Proximal Operator Machinescited by 14
  33. 2020Semi-Supervised Learning with Meta-Gradientcited by 7
  34. 2021Optimization Induced Equilibrium Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 15; 2024: 15; 2025: 16; 2026: 15.

75 papers.

Sort:

By area:

  1. 2025Hierarchical Reasoning Modelcited by 177
  2. 2022Deep Equilibrium Optical Flow Estimationcited by 64
  3. 2023One-Step Diffusion Distillation via Deep Equilibrium Modelscited by 61
  4. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  5. 2022Online Training Through Time for Spiking Neural Networkscited by 49
  6. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 47
  7. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  8. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  9. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 38
  10. 2023Leveraging sparse and shared feature activations for disentangled representation learningcited by 32
  11. 2022Optimization-Induced Graph Implicit Nonlinear Diffusioncited by 28
  12. 2023One-step differentiation of iterative algorithmscited by 28
38 more
  1. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 28
  2. 2023Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Modelscited by 27
  3. 2022Eliminating Gradient Conflict in Reference-based Line-Art Colorizationcited by 25
  4. 2022Alternating Differentiation for Optimization Layerscited by 22
  5. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 22
  6. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  7. 2022MGNNI: Multiscale Graph Neural Networks with Implicit Layerscited by 19
  8. 2023Deep Equilibrium Based Neural Operators for Steady-State PDEscited by 19
  9. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 18
  10. 2023Deep Equilibrium Diffusion Restoration with Parallel Samplingcited by 17
  11. 2024Fixed Point Diffusion Modelscited by 16
  12. 2022Explainable AI via Learning to Optimizecited by 14
  13. 2023A Closer Look at the Adversarial Robustness of Deep Equilibrium Modelscited by 13
  14. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 13
  15. 2023Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splittingcited by 11
  16. 2023Deep Equilibrium Object Detectioncited by 10
  17. 2024Differentiation Through Black-Box Quadratic Programming Solverscited by 10
  18. 2024Positive concave deep equilibrium modelscited by 7
  19. 2023Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-netscited by 6
  20. 2024LPGD: A General Framework for Backpropagation through Embedded Optimization Layerscited by 6
  21. 2022Deep Equilibrium Models for Video Snapshot Compressive Imagingcited by 5
  22. 2024Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registrationcited by 5
  23. 2025Reversible Deep Equilibrium Modelscited by 5
  24. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4
  25. 2025DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flowscited by 4
  26. 2024Efficient and Effective Implicit Dynamic Graph Neural Networkcited by 3
  27. 2024IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networkscited by 3
  28. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  29. 2025PRDP: Progressively Refined Differentiable Physicscited by 3
  30. 2025Self-composing neural operators for high-frequency and multiscale PDE surrogatescited by 3
  31. 2025DEQuify your force field: More efficient simulations using deep equilibrium modelscited by 3
  32. 2025End-to-End Training of High-Dimensional Optimal Control with Implicit Hamiltonians via Jacobian-Free Backpropagationcited by 3
  33. 2023Efficient Training of Deep Equilibrium Modelscited by 2
  34. 2023Improving Adversarial Robustness of DEQs with Explicit Regulations Along the Neural Dynamicscited by 2
  35. 2025Recovering Pulse Waves from Video Using Deep Unrolling and Deep Equilibrium Modelscited by 2
  36. 2025Bridging Classical and Learning-based Iterative Registration through Deep Equilibrium Modelscited by 2
  37. 2026Consistency Deep Equilibrium Modelscited by 2
  38. 2026Energy-guided Recursive Modelcited by 2

50 most cited of 75 All 75 (JSON)