# JFB: Jacobian-Free Backpropagation for Implicit Networks

arXiv: https://arxiv.org/abs/2103.12803 · HTML: https://mlsanity.com/paper/2103.12803/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 76 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport](https://mlsanity.com/paper/2006.00104/) (2020)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Robust Large Margin Deep Neural Networks](https://mlsanity.com/paper/1605.08254/) (2016)
- [An Introduction to Deep Generative Modeling](https://mlsanity.com/paper/2103.05180/) (2021)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019)
- [ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs](https://mlsanity.com/paper/1902.10298/) (2019)
- [Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters](https://mlsanity.com/paper/1511.06727/) (2015)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [Reviving and Improving Recurrent Back-Propagation](https://mlsanity.com/paper/1803.06396/) (2018)
- [Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows](https://mlsanity.com/paper/2005.13420/) (2020)

## Built on by (most cited first, 30 of 86)

- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [A Survey of Contextual Optimization Methods for Decision Making under Uncertainty](https://mlsanity.com/paper/2306.10374/) (2023)
- [Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modeling](https://mlsanity.com/paper/2203.08205/) (2022)
- [Online Training Through Time for Spiking Neural Networks](https://mlsanity.com/paper/2210.04195/) (2022)
- [Learned reconstruction methods with convergence guarantees](https://mlsanity.com/paper/2206.05431/) (2022)
- [A Neural-Network-Based Convex Regularizer for Inverse Problems](https://mlsanity.com/paper/2211.12461/) (2022)
- [Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers](https://mlsanity.com/paper/2606.18206/) (2026)
- [Online Deep Equilibrium Learning for Regularization by Denoising](https://mlsanity.com/paper/2205.13051/) (2022)
- [Alternating Differentiation for Optimization Layers](https://mlsanity.com/paper/2210.01802/) (2022)
- [Deep Equilibrium Based Neural Operators for Steady-State PDEs](https://mlsanity.com/paper/2312.00234/) (2023)
- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation](https://mlsanity.com/paper/2111.12193/) (2021)
- [Making Scalable Meta Learning Practical](https://mlsanity.com/paper/2310.05674/) (2023)
- [Unlocking Slot Attention by Changing Optimal Transport Costs](https://mlsanity.com/paper/2301.13197/) (2023)
- [Fixed Point Diffusion Models](https://mlsanity.com/paper/2401.08741/) (2024)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [Explainable AI via Learning to Optimize](https://mlsanity.com/paper/2204.14174/) (2022)
- [Self-conditioned Flow Map Language Models via Fixed-point Flows](https://mlsanity.com/paper/2607.00714/) (2026)
- [Solve the Loop: Attractor Models for Language and Reasoning](https://mlsanity.com/paper/2605.12466/) (2026)
- [Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splitting](https://mlsanity.com/paper/2301.13395/) (2023)
- [Deep Equilibrium Object Detection](https://mlsanity.com/paper/2308.09564/) (2023)
- [Differentiation Through Black-Box Quadratic Programming Solvers](https://mlsanity.com/paper/2410.06324/) (2024)
- [On Logical Extrapolation for Mazes with Recurrent and Implicit Networks](https://mlsanity.com/paper/2410.03020/) (2024)
- [Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descent](https://mlsanity.com/paper/2507.11461/) (2025)
- [Model-corrected learned primal-dual models for fast limited-view photoacoustic tomography](https://mlsanity.com/paper/2304.01963/) (2023)
- [Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures](https://mlsanity.com/paper/2402.02697/) (2024)
- [Continuous Deep Equilibrium Models: Training Neural ODEs faster by integrating them to Infinity](https://mlsanity.com/paper/2201.12240/) (2022)
- [Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training](https://mlsanity.com/paper/2206.07875/) (2022)
- [Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration](https://mlsanity.com/paper/2406.07361/) (2024)
- [Reversible Deep Equilibrium Models](https://mlsanity.com/paper/2509.12917/) (2025)

## Abstract (arXiv)

A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences. Solving for the fixed point varies in complexity, depending on provided data and an error tolerance. Importantly, implicit networks may be trained with fixed memory costs in stark contrast to feedforward networks, whose memory requirements scale linearly with depth. However, there is no free lunch -- backpropagation through implicit networks often requires solving a costly Jacobian-based equation arising from the implicit function theorem. We propose Jacobian-Free Backpropagation (JFB), a fixed-memory approach that circumvents the need to solve Jacobian-based equations. JFB makes implicit networks faster to train and significantly easier to implement, without sacrificing test accuracy. Our experiments show implicit networks trained with JFB are competitive with feedforward networks and prior implicit networks given the same number of parameters.