From Learning to Optimize to Learning Optimization Algorithms
Not on the map. Why?
Abstract (arXiv)
Towards designing learned optimization algorithms that are usable beyond their training setting, we identify key principles that classical algorithms obey, but have up to now, not been used for Learning to Optimize (L2O). Following these principles, we provide a general design pipeline, taking into account data, architecture and learning strategy, and thereby enabling a synergy between classical optimization and L2O, resulting in a philosophy of Learning Optimization Algorithms. As a consequence our learned algorithms perform well far beyond problems from the training distribution. We demonstrate the success of these novel principles by designing a new learning-enhanced BFGS algorithm and provide numerical experiments evidencing its adaptation to many settings at test time.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 46 of its 83 references that are in the map. First posted on arXiv 28 May 2024.
Builds on
46 of 83 references.
References · 46
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2017Deep Setscited by 1,394
- 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2016Learning to learn by gradient descent by gradient descentcited by 616
- 2020Plug-and-Play Image Restoration with Deep Denoiser Priorcited by 467
- 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
- 2016Learning to Optimizecited by 195
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 158
- 2017Learned Optimizers that Scale and Generalizecited by 109
- 2017Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problemscited by 107
34 more
- 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
- 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
- 2021Equivariant Imaging: Learning Beyond the Range Spacecited by 83
- 2022Tutorial on amortized optimizationcited by 83
- 2020Group Equivariant Stand-Alone Self-Attention For Visioncited by 63
- 2014Global convergence of the Heavy-ball method for convex optimizationcited by 60
- 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
- 2022Towards understanding how momentum improves generalization in deep learningcited by 40
- 2022Imaging with Equivariant Deep Learningcited by 35
- 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
- 2023Learning to Warm-Start Fixed-Point Optimization Algorithmscited by 32
- 2020Training Stronger Baselines for Learning to Optimizecited by 31
- 2019Learning step sizes for unfolded sparse codingcited by 29
- 2023Equivariant plug-and-play image reconstructioncited by 26
- 2020Safeguarded Learned Convex Optimizationcited by 24
- 2019First-order optimization algorithms via inertial systems with Hessian driven dampingcited by 22
- 2020Greedy Quasi-Newton Methods with Explicit Superlinear Convergencecited by 21
- 2023Optimization Dynamics of Equivariant and Augmented Neural Networkscited by 17
- 2016Fast convex optimization via inertial dynamics with Hessian driven dampingcited by 13
- 2022Transformer-Based Learned Optimizationcited by 13
- 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
- 2019Controlling Neural Networks via Energy Dissipationcited by 12
- 2023What functions can Graph Neural Networks compute on random graphs? The role of Positional Encodingcited by 11
- 2020Guarantees for Tuning the Step Size using a Learning-to-Learn Approachcited by 10
- 2018On Quasi-Newton Forward--Backward Splitting: Proximal Calculus and Convergencecited by 8
- 2023Any-dimensional equivariant neural networkscited by 8
- 2022Sharpened Quasi-Newton Methods: Faster Superlinear Rate and Larger Local Convergence Neighborhoodcited by 7
- 2024Learning to optimize with convergence guarantees using nonlinear system theorycited by 7
- 2021Unrolled Variational Bayesian Algorithm for Image Blind Deconvolutioncited by 4
- 2022PAC-Bayesian Learning of Optimization Algorithmscited by 4
- 2022Learning to Optimize Quasi-Newton Methodscited by 3
- 2023Continuous Newton-like Methods featuring Inertia and Variable Masscited by 2
- 2023Boosting Data-Driven Mirror Descent with Randomization, Equivariance, and Accelerationcited by 2
- 2022Inertial Quasi-Newton Methods for Monotone Inclusion: Efficient Resolvent Calculus and Primal-Dual Methodscited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.