# Towards Constituting Mathematical Structures for Learning to Optimize

arXiv: https://arxiv.org/abs/2305.18577 · HTML: https://mlsanity.com/paper/2305.18577/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 60 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Learned Primal-dual Reconstruction](https://mlsanity.com/paper/1707.06474/) (2017)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [Formal Mathematics Statement Curriculum Learning](https://mlsanity.com/paper/2202.01344/) (2022)
- [Online Learning Rate Adaptation with Hypergradient Descent](https://mlsanity.com/paper/1703.04782/) (2017)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems](https://mlsanity.com/paper/1704.03488/) (2017)
- [Understanding and correcting pathologies in the training of learned optimizers](https://mlsanity.com/paper/1810.10180/) (2018)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [CNN-Based Projected Gradient Descent for Consistent Image Reconstruction](https://mlsanity.com/paper/1709.01809/) (2017)
- [Understanding Short-Horizon Bias in Stochastic Meta-Optimization](https://mlsanity.com/paper/1803.02021/) (2018)
- [Provable Guarantees for Gradient-Based Meta-Learning](https://mlsanity.com/paper/1902.10644/) (2019)
- [Learned D-AMP: Principled Neural Network based Compressive Image Recovery](https://mlsanity.com/paper/1704.06625/) (2017)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves](https://mlsanity.com/paper/2009.11243/) (2020)
- [Learning Gradient Descent: Better Generalization and Longer Horizons](https://mlsanity.com/paper/1703.03633/) (2017)
- [Learning-to-Learn Stochastic Gradient Descent with Biased Regularization](https://mlsanity.com/paper/1903.10399/) (2019)
- [Neural Proximal Gradient Descent for Compressive Imaging](https://mlsanity.com/paper/1806.03963/) (2018)
- [Deep Unfolded Robust PCA with Application to Clutter Suppression in Ultrasound](https://mlsanity.com/paper/1811.08252/) (2018)
- [The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning](https://mlsanity.com/paper/2008.10857/) (2020)
- [Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection](https://mlsanity.com/paper/2110.05649/) (2021)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [Training Stronger Baselines for Learning to Optimize](https://mlsanity.com/paper/2010.09089/) (2020)
- [Learning step sizes for unfolded sparse coding](https://mlsanity.com/paper/1905.11071/) (2019)

## Built on by (most cited first, 13 of 13)

- [ODE-based Learning to Optimize](https://mlsanity.com/paper/2406.02006/) (2024)
- [Deep FlexQP: Accelerated Nonlinear Programming via Deep Unfolding](https://mlsanity.com/paper/2512.01565/) (2025)
- [From Learning to Optimize to Learning Optimization Algorithms](https://mlsanity.com/paper/2405.18222/) (2024)
- [A Generalization Result for Convergence in Learning-to-Optimize](https://mlsanity.com/paper/2410.07704/) (2024)
- [Accelerating Optimization via Differentiable Stopping Time](https://mlsanity.com/paper/2505.22509/) (2025)
- [A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods](https://mlsanity.com/paper/2412.00059/) (2024)
- [Learning Provably Improves the Convergence of Gradient Descent](https://mlsanity.com/paper/2501.18092/) (2025)
- [Towards Robust Learning to Optimize with Theoretical Guarantees](https://mlsanity.com/paper/2506.14263/) (2025)
- [Bundle Network: a Machine Learning-Based Bundle Method](https://mlsanity.com/paper/2509.24736/) (2025)
- [When Descent Is Too Stable: Event-Triggered Hamiltonian Learning to Optimize](https://mlsanity.com/paper/2605.06868/) (2026)
- [Learning to Optimize Radiotherapy Plans via Fluence Maps Diffusion Model Generation and LSTM-based Optimization](https://mlsanity.com/paper/2605.13713/) (2026)
- [SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining](https://mlsanity.com/paper/2609.34681/) (2026)
- [Learned Preconditioning for a Primal-Dual Interior-Point Method](https://mlsanity.com/paper/2609.35665/) (2026)

## Abstract (arXiv)

Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years. A generic L2O approach parameterizes the iterative update rule and learns the update direction as a black-box network. While the generic approach is widely applicable, the learned model can overfit and may not generalize well to out-of-distribution test sets. In this paper, we derive the basic mathematical conditions that successful update rules commonly satisfy. Consequently, we propose a novel L2O model with a mathematics-inspired structure that is broadly applicable and generalized well to out-of-distribution problems. Numerical simulations validate our theoretical findings and demonstrate the superior empirical performance of the proposed L2O model.