mlsanityData to 14 Oct 2026

Automatic Differentiation of Optimization Algorithms with Time-Varying Updates

arXiv 2410.15923 ↗First posted 21 Oct 2024math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Numerous Optimization Algorithms have a time-varying update rule thanks to, for instance, a changing step size, momentum parameter or, Hessian approximation. In this paper, we apply unrolled or automatic differentiation to a time-varying iterative process and provide convergence (rate) guarantees for the resulting derivative iterates. We adapt these convergence results and apply them to proximal gradient descent with variable step size and FISTA when solving partly smooth problems. We confirm our findings numerically by solving ℓ1 and ℓ2-regularized linear and logisitc regression respectively. Our theoretical and numerical results show that the convergence rate of the algorithm is reflected in its derivative iterates.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 61 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

27 of 61 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  3. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  4. 2015Automatic differentiation in machine learning: a surveycited by 855
  5. 2019Deep Equilibrium Modelscited by 615
  6. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  7. 2019Differentiable Convex Optimization Layerscited by 465
  8. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  9. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  10. 2021Efficient and Modular Implicit Differentiationcited by 203
  11. 2020Multiscale Deep Equilibrium Modelscited by 151
  12. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
15 more
  1. 2020Monotone operator equilibrium networkscited by 100
  2. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  3. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  4. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
  5. 2020A mathematical model for automatic differentiation in machine learningcited by 45
  6. 2023One-step differentiation of iterative algorithmscited by 28
  7. 2022Automatic differentiation of nonsmooth iterative algorithmscited by 25
  8. 2020Super-efficiency of automatic differentiation for functions defined as a minimumcited by 24
  9. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  10. 2019Automatic Differentiation of Some First-Order Methods in Parametric Optimizationcited by 18
  11. 2022TorchOpt: An Efficient Library for Differentiable Optimizationcited by 17
  12. 2022The Curse of Unrolling: Rate of Differentiating Through Optimizationcited by 16
  13. 2014Local Linear Convergence of Forward-Backward under Partial Smoothnesscited by 13
  14. 2022The derivatives of Sinkhorn-Knopp convergecited by 9
  15. 2022Fixed-Point Automatic Differentiation of Forward--Backward Splitting Algorithms for Partly Smooth Functionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Understanding the Curse of Unrollingcited by 0