# Automatic Differentiation of Some First-Order Methods in Parametric Optimization

arXiv: https://arxiv.org/abs/1910.05696 · HTML: https://mlsanity.com/paper/1910.05696/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2019.

## Built on by (most cited first, 18 of 18)

- [One-step differentiation of iterative algorithms](https://mlsanity.com/paper/2305.13768/) (2023)
- [Automatic differentiation of nonsmooth iterative algorithms](https://mlsanity.com/paper/2206.00457/) (2022)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [Continuation Path Learning for Homotopy Optimization](https://mlsanity.com/paper/2307.12551/) (2023)
- [The derivatives of Sinkhorn-Knopp converge](https://mlsanity.com/paper/2207.12717/) (2022)
- [Analyzing Inexact Hypergradients for Bilevel Learning](https://mlsanity.com/paper/2301.04764/) (2023)
- [Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems](https://mlsanity.com/paper/2212.07844/) (2022)
- [On the complexity of nonsmooth automatic differentiation](https://mlsanity.com/paper/2206.01730/) (2022)
- [Fixed-Point Automatic Differentiation of Forward--Backward Splitting Algorithms for Partly Smooth Functions](https://mlsanity.com/paper/2208.03107/) (2022)
- [An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning](https://mlsanity.com/paper/2308.10098/) (2023)
- [A primal-dual algorithm for image reconstruction with input-convex neural network regularizers](https://mlsanity.com/paper/2410.12441/) (2024)
- [An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation](https://mlsanity.com/paper/2412.06436/) (2024)
- [Bilevel Learning with Inexact Stochastic Gradients](https://mlsanity.com/paper/2412.12049/) (2024)
- [Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization](https://mlsanity.com/paper/2402.16748/) (2024)
- [Automatic Differentiation of Optimization Algorithms with Time-Varying Updates](https://mlsanity.com/paper/2410.15923/) (2024)
- [Derivatives of Stochastic Gradient Descent in parametric optimization](https://mlsanity.com/paper/2405.15894/) (2024)
- [Bilevel Learning via Inexact Stochastic Gradient Descent](https://mlsanity.com/paper/2511.06774/) (2025)
- [Understanding the Curse of Unrolling](https://mlsanity.com/paper/2602.19733/) (2026)

## Abstract (arXiv)

We aim at computing the derivative of the solution to a parametric optimization problem with respect to the involved parameters. For a class broader than that of strongly convex functions, this can be achieved by automatic differentiation of iterative minimization algorithms. If the iterative algorithm converges pointwise, then we prove that the derivative sequence also converges pointwise to the derivative of the minimizer with respect to the parameters. Moreover, we provide convergence rates for both sequences. In particular, we prove that the accelerated convergence rate of the Heavy-ball method compared to Gradient Descent also accelerates the derivative computation. An experiment with L2-Regularized Logistic Regression validates the theoretical results.