# One-step differentiation of iterative algorithms

arXiv: https://arxiv.org/abs/2305.13768 · HTML: https://mlsanity.com/paper/2305.13768/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Truncated Back-propagation for Bilevel Optimization](https://mlsanity.com/paper/1810.10667/) (2018)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Implicit Graph Neural Networks](https://mlsanity.com/paper/2009.06211/) (2020)
- [A framework for bilevel optimization that enables stochastic and global variance reduction algorithms](https://mlsanity.com/paper/2201.13409/) (2022)
- [Is Attention Better Than Matrix Decomposition?](https://mlsanity.com/paper/2109.04553/) (2021)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters](https://mlsanity.com/paper/1511.06727/) (2015)
- [Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance](https://mlsanity.com/paper/1805.11897/) (2018)
- [Nonsmooth Implicit Differentiation for Machine Learning and Optimization](https://mlsanity.com/paper/2106.04350/) (2021)
- [Implicit differentiation of Lasso-type models for hyperparameter optimization](https://mlsanity.com/paper/2002.08943/) (2020)
- [Backpropagation through Combinatorial Algorithms: Identity with Projection Works](https://mlsanity.com/paper/2205.15213/) (2022)
- [SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models](https://mlsanity.com/paper/2106.00553/) (2021)
- [Automatic differentiation of nonsmooth iterative algorithms](https://mlsanity.com/paper/2206.00457/) (2022)
- [Super-efficiency of automatic differentiation for functions defined as a minimum](https://mlsanity.com/paper/2002.03722/) (2020)
- [Automatic Differentiation of Some First-Order Methods in Parametric Optimization](https://mlsanity.com/paper/1910.05696/) (2019)
- [The Curse of Unrolling: Rate of Differentiating Through Optimization](https://mlsanity.com/paper/2209.13271/) (2022)
- [The derivatives of Sinkhorn-Knopp converge](https://mlsanity.com/paper/2207.12717/) (2022)
- [Analyzing Inexact Hypergradients for Bilevel Learning](https://mlsanity.com/paper/2301.04764/) (2023)

## Built on by (most cited first, 28 of 28)

- [Implicit Diffusion: Efficient Optimization through Stochastic Sampling](https://mlsanity.com/paper/2402.05468/) (2024)
- [Functional Bilevel Optimization for Machine Learning](https://mlsanity.com/paper/2403.20233/) (2024)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [Let Go of Your Labels with Unsupervised Transfer](https://mlsanity.com/paper/2406.07236/) (2024)
- [Differentiation Through Black-Box Quadratic Programming Solvers](https://mlsanity.com/paper/2410.06324/) (2024)
- [The quest for the GRAph Level autoEncoder (GRALE)](https://mlsanity.com/paper/2505.22109/) (2025)
- [On Logical Extrapolation for Mazes with Recurrent and Implicit Networks](https://mlsanity.com/paper/2410.03020/) (2024)
- [Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descent](https://mlsanity.com/paper/2507.11461/) (2025)
- [Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling](https://mlsanity.com/paper/2410.01440/) (2024)
- [A primal-dual algorithm for image reconstruction with input-convex neural network regularizers](https://mlsanity.com/paper/2410.12441/) (2024)
- [PRDP: Progressively Refined Differentiable Physics](https://mlsanity.com/paper/2502.19611/) (2025)
- [End-to-End Training of High-Dimensional Optimal Control with Implicit Hamiltonians via Jacobian-Free Backpropagation](https://mlsanity.com/paper/2510.00359/) (2025)
- [A Fully First-Order Layer for Differentiable Optimization](https://mlsanity.com/paper/2512.02494/) (2025)
- [Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting](https://mlsanity.com/paper/2603.08118/) (2026)
- [Inexact subgradient methods for semialgebraic functions](https://mlsanity.com/paper/2404.19517/) (2024)
- [Automatic Differentiation of Optimization Algorithms with Time-Varying Updates](https://mlsanity.com/paper/2410.15923/) (2024)
- [Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmax](https://mlsanity.com/paper/2506.10572/) (2025)
- [On the Convergence of Jacobian-Free Backpropagation for Optimal Control Problems with Implicit Hamiltonians](https://mlsanity.com/paper/2602.00921/) (2026)
- [Derivatives of Stochastic Gradient Descent in parametric optimization](https://mlsanity.com/paper/2405.15894/) (2024)
- [You Only Look One Step: Accelerating Backpropagation in Diffusion Sampling with Gradient Shortcuts](https://mlsanity.com/paper/2505.07477/) (2025)
- [Differentiable Expectation-Maximisation and Applications to Gaussian Mixture Model Optimal Transport](https://mlsanity.com/paper/2509.02109/) (2025)
- [Bilevel Learning via Inexact Stochastic Gradient Descent](https://mlsanity.com/paper/2511.06774/) (2025)
- [Optimal Transport under Group Fairness Constraints](https://mlsanity.com/paper/2601.07144/) (2026)
- [Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Run](https://mlsanity.com/paper/2605.27292/) (2026)
- [A Geometric View of SRC: Learning Representations for Stable Residual Inference](https://mlsanity.com/paper/2605.29673/) (2026)
- [A geometry-based deep equilibrium model for image restoration under multiplicative Gamma noise](https://mlsanity.com/paper/2608.04944/) (2026)
- [Scalable Self-Supervised Learning for Multiphase AC-OPF in Distribution Systems with Topology Reconfiguration](https://mlsanity.com/paper/2608.25095/) (2026)
- [CLEAR: Complex Learned Explicit Analytical Regularization for Ultra-Accelerated 4D Flow CMR Reconstruction](https://mlsanity.com/paper/2609.22950/) (2026)

## Abstract (arXiv)

In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as performant as implicit differentiation for fast algorithms (e.g., superlinear optimization methods). We provide a complete theoretical approximation analysis with specific examples (Newton's method, gradient descent) along with its consequences in bilevel optimization. Several numerical examples illustrate the well-foundness of the one-step estimator.