mlsanityData to 7 Oct 2026

Learning to Optimize: A Primer and A Benchmark

arXiv 2103.12828 ↗First posted 23 Mar 2021math.OC cs.LG stat.MLcited by 206

Not on the map. Why?

Abstract (arXiv)

Learning to optimize (L2O) is an emerging approach that leverages machine learning to develop optimization methods, aiming at reducing the laborious iterations of hand engineering. It automates the design of an optimization method based on its performance on a set of training problems. This data-driven procedure generates methods that can efficiently solve problems similar to those in the training. In sharp contrast, the typical and traditional designs of optimization methods are theory-driven, so they obtain performance guarantees over the classes of problems specified by the theory. The difference makes L2O suitable for repeatedly solving a certain type of optimization problems over a specific distribution of data, while it typically fails on out-of-distribution problems. The practicality of L2O depends on the type of target optimization, the chosen architecture of the method to learn, and the training procedure. This new paradigm has motivated a community of researchers to explore L2O and report their findings. This article is poised to be the first comprehensive survey and benchmark of L2O for continuous optimization. We set up taxonomies, categorize existing works and research directions, present insights, and identify open challenges. We also benchmarked many existing L2O approaches on a few but representative optimization problems. For reproducible research and fair benchmarking purposes, we released our software implementation and data in the package Open-L2O at https://github.com/VITA-Group/Open-L2O.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Builds on 150 of its 280 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

150 of 280 references.

References · 150

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2016Learning to learn by gradient descent by gradient descentcited by 616
  4. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 612
  5. 2017Learning Combinatorial Optimization Algorithms over Graphscited by 557
  6. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
  7. 2017Learning Deep CNN Denoiser Prior for Image Restorationcited by 407
  8. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  9. 2017MoDL: Model Based Deep Learning Architecture for Inverse Problemscited by 384
  10. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  11. 2017Compressed Sensing using Generative Modelscited by 337
  12. 2017PDE-Net: Learning PDEs from Datacited by 304
38 more
  1. 2017Learned Primal-dual Reconstructioncited by 251
  2. 2016Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applicationscited by 244
  3. 2015Conditional Random Fields as Recurrent Neural Networkscited by 213
  4. 2018Sorting out Lipschitz function approximationcited by 211
  5. 2018PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Networkcited by 208
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  7. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 197
  8. 2016Learning to Optimizecited by 195
  9. 2015Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restorationcited by 194
  10. 2017The Case for Learned Index Structurescited by 188
  11. 2018Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Searchcited by 182
  12. 2017Solving ill-posed inverse problems using iterative deep neural networkscited by 145
  13. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  14. 2020Model-Based Deep Learningcited by 136
  15. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  16. 2020A Model-driven Deep Neural Network for Single Image Rain Removalcited by 125
  17. 2019Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernelscited by 112
  18. 2017Learned Optimizers that Scale and Generalizecited by 109
  19. 2017Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problemscited by 107
  20. 2018Adversarial Regularizers in Inverse Problemscited by 102
  21. 2018Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flowcited by 97
  22. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  23. 2018NETT: Solving Inverse Problems with Deep Neural Networkscited by 89
  24. 2019DeepRED: Deep Image Prior Powered by REDcited by 84
  25. 2018Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholdscited by 83
  26. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  27. 2018Denoising Prior Driven Deep Neural Network for Image Restorationcited by 80
  28. 2019Learning to Optimize Multigrid PDE Solverscited by 79
  29. 2015Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolationcited by 78
  30. 2020Plug-and-Play Algorithms for Large-scale Snapshot Compressive Imagingcited by 78
  31. 2014Deep Unfolding: Model-Based Inspiration of Novel Deep Architecturescited by 76
  32. 2017CNN-Based Projected Gradient Descent for Consistent Image Reconstructioncited by 75
  33. 2019Plug and play methods for magnetic resonance imaging (long version)cited by 75
  34. 2018An Online Plug-and-Play Algorithm for Regularized Image Reconstructioncited by 70
  35. 2017One Network to Solve Them All --- Solving Linear Inverse Problems using Deep Projection Modelscited by 67
  36. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 66
  37. 2017Learned D-AMP: Principled Neural Network based Compressive Image Recoverycited by 65
  38. 2019A Model for Learned Bloom Filters, and Optimizing by Sandwichingcited by 64

50 most cited of 150 All 150 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 36; 2023: 35; 2024: 46; 2025: 44; 2026: 33.

206 papers.

Sort:

By area:

  1. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  2. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  3. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  4. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  5. 2022Tutorial on amortized optimizationcited by 83
  6. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  7. 2022Self-Supervised Primal-Dual Learning for Constrained Optimizationcited by 60
  8. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  9. 2023Algorithm Evolution Using Large Language Modelcited by 52
  10. 2022Model-Based Deep Learning: On the Intersection of Deep Learning and Optimizationcited by 50
  11. 2022Equivariant Hypergraph Diffusion Neural Operatorscited by 45
  12. 2026EvoX: Meta-Evolution for Automated Discoverycited by 45
38 more
  1. 2024Meta Flow Matching: Integrating Vector Fields on the Wasserstein Manifoldcited by 41
  2. 2022On Representing Linear Programs by Graph Neural Networkscited by 39
  3. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  4. 2022Automated Dynamic Algorithm Configurationcited by 37
  5. 2023Learning to Warm-Start Fixed-Point Optimization Algorithmscited by 32
  6. 2024Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?cited by 31
  7. 2022On amortizing convex conjugates for optimal transportcited by 30
  8. 2022Meta Optimal Transportcited by 28
  9. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  10. 2021Learning to Learn Graph Topologiescited by 26
  11. 2023What's in a Prior? Learned Proximal Networks for Inverse Problemscited by 24
  12. 2024Scale Equivariant Graph Metanetworkscited by 23
  13. 2021Hyperparameter Tuning is All You Need for LISTAcited by 22
  14. 2024Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Modelscited by 22
  15. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  16. 2023Improving physics-informed neural networks with meta-learned optimizationcited by 19
  17. 2023Machine Learning for Large-Scale Optimization in 6G Wireless Networkscited by 18
  18. 2023Neural incomplete factorization: learning preconditioners for the conjugate gradient methodcited by 18
  19. 2021Feasibility-based Fixed Point Networkscited by 17
  20. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
  21. 2022Landscape Learning for Neural Network Inversioncited by 15
  22. 2023B2Opt: Learning to Optimize Black-box Optimization with Little Budgetcited by 15
  23. 2022Explainable AI via Learning to Optimizecited by 14
  24. 2024PDHG-Unrolled Learning-to-Optimize Method for Large-Scale Linear Programmingcited by 14
  25. 2022Transformer-Based Learned Optimizationcited by 13
  26. 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
  27. 2025Planning of Heuristics: Strategic Planning on Large Language Models with Monte Carlo Tree Search for Automating Heuristic Optimizationcited by 13
  28. 2022A Deep Conjugate Direction Method for Iteratively Solving Linear Systemscited by 12
  29. 2023Continuation Path Learning for Homotopy Optimizationcited by 12
  30. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 12
  31. 2023Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splittingcited by 11
  32. 2024Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Processcited by 11
  33. 2024From Large Language Models and Optimization to Decision Optimization CoPilot: A Research Manifestocited by 11
  34. 2024Gradient Methods with Online Scalingcited by 11
  35. 2024Convergence Rate Analysis of LIONcited by 11
  36. 2022Turbocharging Solution Concepts: Solving NEs, CEs and CCEs with Neural Equilibrium Solverscited by 10
  37. 2022Learning to Optimize in Model Predictive Controlcited by 10
  38. 2023Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedbackcited by 10

50 most cited of 206 All 206 (JSON)