# The Curse of Unrolling: Rate of Differentiating Through Optimization

arXiv: https://arxiv.org/abs/2209.13271 · HTML: https://mlsanity.com/paper/2209.13271/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 27 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [Hyperparameter optimization with approximate gradient](https://mlsanity.com/paper/1602.02355/) (2016)
- [Unrolled Generative Adversarial Networks](https://mlsanity.com/paper/1611.02163/) (2016)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver](https://mlsanity.com/paper/1905.12149/) (2019)
- [On the Iteration Complexity of Hypergradient Computation](https://mlsanity.com/paper/2006.16218/) (2020)
- [Structured Attention Networks](https://mlsanity.com/paper/1702.00887/) (2017)
- [Connecting Generative Adversarial Networks and Actor-Critic Methods](https://mlsanity.com/paper/1610.01945/) (2016)
- [Acceleration Methods](https://mlsanity.com/paper/2101.09545/) (2021)
- [Implicit differentiation of Lasso-type models for hyperparameter optimization](https://mlsanity.com/paper/2002.08943/) (2020)
- [Super-efficiency of automatic differentiation for functions defined as a minimum](https://mlsanity.com/paper/2002.03722/) (2020)
- [Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis](https://mlsanity.com/paper/2006.04299/) (2020)
- [Acceleration via Fractal Learning Rate Schedules](https://mlsanity.com/paper/2103.01338/) (2021)
- [Universal Average-Case Optimality of Polyak Momentum](https://mlsanity.com/paper/2002.04664/) (2020)
- [Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime](https://mlsanity.com/paper/2206.09901/) (2022)
- [Average-case Acceleration Through Spectral Density Estimation](https://mlsanity.com/paper/2002.04756/) (2020)

## Built on by (most cited first, 16 of 16)

- [One-step differentiation of iterative algorithms](https://mlsanity.com/paper/2305.13768/) (2023)
- [FSNet: Feasibility-Seeking Neural Network for Constrained Optimization with Guarantees](https://mlsanity.com/paper/2506.00362/) (2025)
- [Let Go of Your Labels with Unsupervised Transfer](https://mlsanity.com/paper/2406.07236/) (2024)
- [Differentiation Through Black-Box Quadratic Programming Solvers](https://mlsanity.com/paper/2410.06324/) (2024)
- [Deep FlexQP: Accelerated Nonlinear Programming via Deep Unfolding](https://mlsanity.com/paper/2512.01565/) (2025)
- [PRDP: Progressively Refined Differentiable Physics](https://mlsanity.com/paper/2502.19611/) (2025)
- [Bi-Level Policy Optimization with Nystr"om Hypergradients](https://mlsanity.com/paper/2505.11714/) (2025)
- [Controlling Transient Amplification Improves Long-horizon Rollouts](https://mlsanity.com/paper/2605.08856/) (2026)
- [Compact Optimality Verification for Optimization Proxies](https://mlsanity.com/paper/2405.21023/) (2024)
- [TurboMPC: Fast, Scalable, and Differentiable Model Predictive Control on the GPU](https://mlsanity.com/paper/2606.24039/) (2026)
- [Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning](https://mlsanity.com/paper/2403.00865/) (2024)
- [Automatic Differentiation of Optimization Algorithms with Time-Varying Updates](https://mlsanity.com/paper/2410.15923/) (2024)
- [DiffOP: Reinforcement Learning of Optimization-Based Control Policies via Implicit Policy Gradients](https://mlsanity.com/paper/2411.07484/) (2024)
- [Autotuning Bipedal Locomotion MPC with GRFM-Net for Efficient Sim-to-Real Transfer](https://mlsanity.com/paper/2409.15710/) (2024)
- [Bundle Network: a Machine Learning-Based Bundle Method](https://mlsanity.com/paper/2509.24736/) (2025)
- [Understanding the Curse of Unrolling](https://mlsanity.com/paper/2602.19733/) (2026)

## Abstract (arXiv)

Computing the Jacobian of the solution of an optimization problem is a central problem in machine learning, with applications in hyperparameter optimization, meta-learning, optimization as a layer, and dataset distillation, to name a few. Unrolled differentiation is a popular heuristic that approximates the solution using an iterative solver and differentiates it through the computational path. This work provides a non-asymptotic convergence-rate analysis of this approach on quadratic objectives for gradient descent and the Chebyshev method. We show that to ensure convergence of the Jacobian, we can either 1) choose a large learning rate leading to a fast asymptotic convergence but accept that the algorithm may have an arbitrarily long burn-in phase or 2) choose a smaller learning rate leading to an immediate but slower convergence. We refer to this phenomenon as the curse of unrolling. Finally, we discuss open problems relative to this approach, such as deriving a practical update rule for the optimal unrolling strategy and making novel connections with the field of Sobolev orthogonal polynomials.