mlsanityData to 14 Oct 2026

One-step differentiation of iterative algorithms

arXiv 2305.13768 ↗First posted 23 May 2023math.OC cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as performant as implicit differentiation for fast algorithms (e.g., superlinear optimization methods). We provide a complete theoretical approximation analysis with specific examples (Newton's method, gradient descent) along with its consequences in bilevel optimization. Several numerical examples illustrate the well-foundness of the one-step estimator.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

27 of 44 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2021Convex optimizationcited by 2,515
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  5. 2019Deep Equilibrium Modelscited by 615
  6. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  7. 2019Differentiable Convex Optimization Layerscited by 465
  8. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  9. 2021Efficient and Modular Implicit Differentiationcited by 203
  10. 2018Truncated Back-propagation for Bilevel Optimizationcited by 190
  11. 2019Implicit Deep Learningcited by 124
  12. 2020Implicit Graph Neural Networkscited by 100
15 more
  1. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  2. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  3. 2021On Training Implicit Modelscited by 75
  4. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  5. 2018Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distancecited by 56
  6. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  7. 2020Implicit differentiation of Lasso-type models for hyperparameter optimizationcited by 34
  8. 2022Backpropagation through Combinatorial Algorithms: Identity with Projection Workscited by 29
  9. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  10. 2022Automatic differentiation of nonsmooth iterative algorithmscited by 25
  11. 2020Super-efficiency of automatic differentiation for functions defined as a minimumcited by 24
  12. 2019Automatic Differentiation of Some First-Order Methods in Parametric Optimizationcited by 18
  13. 2022The Curse of Unrolling: Rate of Differentiating Through Optimizationcited by 16
  14. 2022The derivatives of Sinkhorn-Knopp convergecited by 9
  15. 2023Analyzing Inexact Hypergradients for Bilevel Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 10; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 24
  2. 2024Functional Bilevel Optimization for Machine Learningcited by 20
  3. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 15
  4. 2024Let Go of Your Labels with Unsupervised Transfercited by 14
  5. 2024Differentiation Through Black-Box Quadratic Programming Solverscited by 10
  6. 2025The quest for the GRAph Level autoEncoder (GRALE)cited by 8
  7. 2024On Logical Extrapolation for Mazes with Recurrent and Implicit Networkscited by 7
  8. 2025Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descentcited by 7
  9. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4
  10. 2024A primal-dual algorithm for image reconstruction with input-convex neural network regularizerscited by 3
  11. 2025PRDP: Progressively Refined Differentiable Physicscited by 3
  12. 2025End-to-End Training of High-Dimensional Optimal Control with Implicit Hamiltonians via Jacobian-Free Backpropagationcited by 3
16 more
  1. 2025A Fully First-Order Layer for Differentiable Optimizationcited by 3
  2. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 3
  3. 2024Inexact subgradient methods for semialgebraic functionscited by 1
  4. 2024Automatic Differentiation of Optimization Algorithms with Time-Varying Updatescited by 1
  5. 2025Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmaxcited by 1
  6. 2026On the Convergence of Jacobian-Free Backpropagation for Optimal Control Problems with Implicit Hamiltonianscited by 1
  7. 2024Derivatives of Stochastic Gradient Descent in parametric optimizationcited by 0
  8. 2025You Only Look One Step: Accelerating Backpropagation in Diffusion Sampling with Gradient Shortcutscited by 0
  9. 2025Differentiable Expectation-Maximisation and Applications to Gaussian Mixture Model Optimal Transportcited by 0
  10. 2025Bilevel Learning via Inexact Stochastic Gradient Descentcited by 0
  11. 2026Optimal Transport under Group Fairness Constraintscited by 0
  12. 2026Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Runcited by 0
  13. 2026A Geometric View of SRC: Learning Representations for Stable Residual Inferencecited by 0
  14. 2026A geometry-based deep equilibrium model for image restoration under multiplicative Gamma noisecited by 0
  15. 2026Scalable Self-Supervised Learning for Multiphase AC-OPF in Distribution Systems with Topology Reconfigurationcited by 0
  16. 2026CLEAR: Complex Learned Explicit Analytical Regularization for Ultra-Accelerated 4D Flow CMR Reconstructioncited by 0