mlsanityData to 7 Oct 2026

Towards Constituting Mathematical Structures for Learning to Optimize

arXiv 2305.18577 ↗First posted 29 May 2023cs.LG math.OC stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years. A generic L2O approach parameterizes the iterative update rule and learns the update direction as a black-box network. While the generic approach is widely applicable, the learned model can overfit and may not generalize well to out-of-distribution test sets. In this paper, we derive the basic mathematical conditions that successful update rules commonly satisfy. Consequently, we propose a novel L2O model with a mathematics-inspired structure that is broadly applicable and generalized well to out-of-distribution problems. Numerical simulations validate our theoretical findings and demonstrate the superior empirical performance of the proposed L2O model.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 41 of its 60 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

41 of 60 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Learning to learn by gradient descent by gradient descentcited by 616
  3. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 612
  4. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
  5. 2017Learning Deep CNN Denoiser Prior for Image Restorationcited by 407
  6. 2017Learned Primal-dual Reconstructioncited by 251
  7. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  8. 2016Learning to Optimizecited by 195
  9. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  10. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  11. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  12. 2017Learned Optimizers that Scale and Generalizecited by 109
29 more
  1. 2017Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problemscited by 107
  2. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  3. 2018Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholdscited by 83
  4. 2017CNN-Based Projected Gradient Descent for Consistent Image Reconstructioncited by 75
  5. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  6. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  7. 2017Learned D-AMP: Principled Neural Network based Compressive Image Recoverycited by 65
  8. 2021Linear algebra with transformerscited by 62
  9. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  10. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  11. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  12. 2018Neural Proximal Gradient Descent for Compressive Imagingcited by 46
  13. 2018Deep Unfolded Robust PCA with Application to Clutter Suppression in Ultrasoundcited by 43
  14. 2020The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuningcited by 40
  15. 2021Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detectioncited by 34
  16. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  17. 2020Training Stronger Baselines for Learning to Optimizecited by 31
  18. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  19. 2016Maximal Sparsity with Deep Networks?cited by 29
  20. 2019Learning step sizes for unfolded sparse codingcited by 29
  21. 2020Ada-LISTA: Learned Solvers Adaptive to Varying Modelscited by 22
  22. 2021Hyperparameter Tuning is All You Need for LISTAcited by 22
  23. 2019Differentiable Linearized ADMMcited by 18
  24. 2015Exact Worst-case Performance of First-order Methods for Composite Convex Optimizationcited by 16
  25. 2018Operator Splitting Performance Estimation: Tight contraction factors and optimal parameter selectioncited by 13
  26. 2018LSALSA: Accelerated Source Separation via Learned Sparse Codingcited by 7
  27. 2020Neurally Augmented ALISTAcited by 5
  28. 2019Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsizecited by 4
  29. 2022Learning Proximal Operators to Discover Multiple Optimacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024ODE-based Learning to Optimizecited by 4
  2. 2025Deep FlexQP: Accelerated Nonlinear Programming via Deep Unfoldingcited by 4
  3. 2024From Learning to Optimize to Learning Optimization Algorithmscited by 2
  4. 2024A Generalization Result for Convergence in Learning-to-Optimizecited by 1
  5. 2025Accelerating Optimization via Differentiable Stopping Timecited by 1
  6. 2024A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methodscited by 0
  7. 2025Learning Provably Improves the Convergence of Gradient Descentcited by 0
  8. 2025Towards Robust Learning to Optimize with Theoretical Guaranteescited by 0
  9. 2025Bundle Network: a Machine Learning-Based Bundle Methodcited by 0
  10. 2026When Descent Is Too Stable: Event-Triggered Hamiltonian Learning to Optimizecited by 0
  11. 2026Learning to Optimize Radiotherapy Plans via Fluence Maps Diffusion Model Generation and LSTM-based Optimizationcited by 0
  12. 2026SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretrainingcited by 0
1 more
  1. 2026Learned Preconditioning for a Primal-Dual Interior-Point Methodcited by 0