mlsanityData to 7 Oct 2026

JFB: Jacobian-Free Backpropagation for Implicit Networks

arXiv 2103.12803 ↗First posted 23 Mar 2021cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences. Solving for the fixed point varies in complexity, depending on provided data and an error tolerance. Importantly, implicit networks may be trained with fixed memory costs in stark contrast to feedforward networks, whose memory requirements scale linearly with depth. However, there is no free lunch -- backpropagation through implicit networks often requires solving a costly Jacobian-based equation arising from the implicit function theorem. We propose Jacobian-Free Backpropagation (JFB), a fixed-memory approach that circumvents the need to solve Jacobian-based equations. JFB makes implicit networks faster to train and significantly easier to implement, without sacrificing test accuracy. Our experiments show implicit networks trained with JFB are competitive with feedforward networks and prior implicit networks given the same number of parameters.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

37 of 76 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2019Deep Equilibrium Modelscited by 603
  6. 2019Meta-Learning with Implicit Gradientscited by 523
  7. 2019Augmented Neural ODEscited by 396
  8. 2017Stable Architectures for Deep Neural Networkscited by 340
  9. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  10. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  11. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  12. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
25 more
  1. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
  2. 2018Sorting out Lipschitz function approximationcited by 211
  3. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  4. 2019Universal Approximation with Deep Narrow Networkscited by 198
  5. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  6. 2020Multiscale Deep Equilibrium Modelscited by 149
  7. 2020OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transportcited by 143
  8. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 126
  9. 2019Implicit Deep Learningcited by 124
  10. 2016Robust Large Margin Deep Neural Networkscited by 117
  11. 2021An Introduction to Deep Generative Modelingcited by 104
  12. 2020Monotone operator equilibrium networkscited by 100
  13. 2019A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problemscited by 95
  14. 2019ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEscited by 83
  15. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  16. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  17. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  18. 2020Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flowscited by 36
  19. 2020Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Gamescited by 35
  20. 2019Adaptively Truncating Backpropagation Through Time to Control Gradient Biascited by 22
  21. 2019Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainabilitycited by 17
  22. 2020Implicitly Defined Layers in Neural Networkscited by 7
  23. 2020Differentiable Implicit Layerscited by 6
  24. 2018Multigrid Optimization for Large-Scale Ptychographic Phase Retrievalcited by 3
  25. 2020Wasserstein-based Projections with Applications to Inverse Problemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 15; 2024: 17; 2025: 15; 2026: 22.

84 papers.

Sort:

By area:

  1. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  2. 2023A Survey of Contextual Optimization Methods for Decision Making under Uncertaintycited by 81
  3. 2022Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modelingcited by 65
  4. 2022Online Training Through Time for Spiking Neural Networkscited by 48
  5. 2022Learned reconstruction methods with convergence guaranteescited by 46
  6. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 40
  7. 2022Online Deep Equilibrium Learning for Regularization by Denoisingcited by 22
  8. 2022Alternating Differentiation for Optimization Layerscited by 22
  9. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  10. 2023Deep Equilibrium Based Neural Operators for Steady-State PDEscited by 19
  11. 2021Multiset-Equivariant Set Prediction with Approximate Implicit Differentiationcited by 17
  12. 2023Making Scalable Meta Learning Practicalcited by 17
38 more
  1. 2023Unlocking Slot Attention by Changing Optimal Transport Costscited by 16
  2. 2022Explainable AI via Learning to Optimizecited by 14
  3. 2024Fixed Point Diffusion Modelscited by 14
  4. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 12
  5. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  6. 2023Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splittingcited by 11
  7. 2026Solve the Loop: Attractor Models for Language and Reasoningcited by 11
  8. 2023Deep Equilibrium Object Detectioncited by 10
  9. 2024Differentiation Through Black-Box Quadratic Programming Solverscited by 10
  10. 2026Self-conditioned Flow Map Language Models via Fixed-point Flowscited by 9
  11. 2024On Logical Extrapolation for Mazes with Recurrent and Implicit Networkscited by 7
  12. 2025Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descentcited by 7
  13. 2023Model-corrected learned primal-dual models for fast limited-view photoacoustic tomographycited by 6
  14. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  15. 2022Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinitycited by 5
  16. 2022Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-trainingcited by 5
  17. 2024Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registrationcited by 5
  18. 2025Reversible Deep Equilibrium Modelscited by 5
  19. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4
  20. 2024A Generalization Bound for a Family of Implicit Networkscited by 4
  21. 2023Towards Vision Transformer Unrolling Fixed-Point Algorithm: a Case Study on Image Restorationcited by 3
  22. 2023Test like you Train in Implicit Deep Learningcited by 3
  23. 2024Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspectivecited by 3
  24. 2025PRDP: Progressively Refined Differentiable Physicscited by 3
  25. 2025TomoSelfDEQ: Self-Supervised Deep Equilibrium Learning for Sparse-Angle CT Reconstructioncited by 3
  26. 2025Self-composing neural operators for high-frequency and multiscale PDE surrogatescited by 3
  27. 2025DEQuify your force field: More efficient simulations using deep equilibrium modelscited by 3
  28. 2025End-to-End Training of High-Dimensional Optimal Control with Implicit Hamiltonians via Jacobian-Free Backpropagationcited by 3
  29. 2026torch-sla: Differentiable Sparse Linear Algebra with Adjoint Solvers and Sparse Tensor Parallelism for PyTorchcited by 3
  30. 2026Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Loopingcited by 3
  31. 2023Improving Adversarial Robustness of DEQs with Explicit Regulations Along the Neural Dynamicscited by 2
  32. 2024A primal-dual algorithm for image reconstruction with input-convex neural network regularizerscited by 2
  33. 2024Semi-Implicit Neural Ordinary Differential Equationscited by 2
  34. 2025Recovering Pulse Waves from Video Using Deep Unrolling and Deep Equilibrium Modelscited by 2
  35. 2025Restarted contractive operators to learn at equilibriumcited by 2
  36. 2025Minimizing Surrogate Losses for Decision-Focused Learning using Differentiable Optimizationcited by 2
  37. 2026Fixed-Point Masked Generative Modelingcited by 2
  38. 2022Self-Supervised Deep Equilibrium Models for Inverse Problems with Theoretical Guaranteescited by 1

50 most cited of 84 All 84 (JSON)