# Training Stronger Baselines for Learning to Optimize

arXiv: https://arxiv.org/abs/2010.09089 · HTML: https://mlsanity.com/paper/2010.09089/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2020.

## Built on by (most cited first, 30 of 31)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Graph Neural Networks for Learning Equivariant Representations of Neural Networks](https://mlsanity.com/paper/2403.12143/) (2024)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection](https://mlsanity.com/paper/2110.05649/) (2021)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Symbolic Learning to Optimize: Towards Interpretability and Scalability](https://mlsanity.com/paper/2203.06578/) (2022)
- [B2Opt: Learning to Optimize Black-box Optimization with Little Budget](https://mlsanity.com/paper/2304.11787/) (2023)
- [Towards Constituting Mathematical Structures for Learning to Optimize](https://mlsanity.com/paper/2305.18577/) (2023)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)
- [Mnemosyne: Learning to Train Transformers with Transformers](https://mlsanity.com/paper/2302.01128/) (2023)
- [Learning to Generalize Provably in Learning to Optimize](https://mlsanity.com/paper/2302.11085/) (2023)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [μLO: Compute-Efficient Meta-Generalization of Learned Optimizers](https://mlsanity.com/paper/2406.00153/) (2024)
- [Metamizer: a versatile neural optimizer for fast and accurate physics simulations](https://mlsanity.com/paper/2410.19746/) (2024)
- [Learning to Optimize with Dynamic Mode Decomposition](https://mlsanity.com/paper/2211.16268/) (2022)
- [Learning for Robust Combinatorial Optimization: Algorithm and Application](https://mlsanity.com/paper/2112.10377/) (2021)
- [When Bioprocess Engineering Meets Machine Learning: A Survey from the Perspective of Automated Bioprocess Development](https://mlsanity.com/paper/2209.01083/) (2022)
- [M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation](https://mlsanity.com/paper/2303.00039/) (2023)
- [Celo: Training Versatile Learned Optimizers on a Compute Diet](https://mlsanity.com/paper/2501.12670/) (2025)
- [Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness](https://mlsanity.com/paper/2110.08256/) (2021)
- [Meta-learning Optimizers for Communication-Efficient Learning](https://mlsanity.com/paper/2312.02204/) (2023)
- [From Learning to Optimize to Learning Optimization Algorithms](https://mlsanity.com/paper/2405.18222/) (2024)
- [Consensus Multiplicative Weights Update: Learning to Learn using Projector-based Game Signatures](https://mlsanity.com/paper/2106.02615/) (2021)
- [Optimizer Amalgamation](https://mlsanity.com/paper/2203.06474/) (2022)
- [Accelerating Optimization via Differentiable Stopping Time](https://mlsanity.com/paper/2505.22509/) (2025)
- [Calibration of Derivative Pricing Models: a Multi-Agent Reinforcement Learning Perspective](https://mlsanity.com/paper/2203.06865/) (2022)
- [Learning Regularizers: Learning Optimizers that can Regularize](https://mlsanity.com/paper/2510.08968/) (2025)
- [Efficient Long-Horizon Learning for Learned Optimization](https://mlsanity.com/paper/2607.06772/) (2026)

## Abstract (arXiv)

Learning to optimize (L2O) has gained increasing attention since classical optimizers require laborious problem-specific design and hyperparameter tuning. However, there is a gap between the practical demand and the achievable performance of existing L2O models. Specifically, those learned optimizers are applicable to only a limited class of problems, and often exhibit instability. With many efforts devoted to designing more sophisticated L2O models, we argue for another orthogonal, under-explored theme: the training techniques for those L2O models. We show that even the simplest L2O model could have been trained much better. We first present a progressive training scheme to gradually increase the optimizer unroll length, to mitigate a well-known L2O dilemma of truncation bias (shorter unrolling) versus gradient explosion (longer unrolling). We further leverage off-policy imitation learning to guide the L2O learning, by taking reference to the behavior of analytical optimizers. Our improved training techniques are plugged into a variety of state-of-the-art L2O models, and immediately boost their performance, without making any change to their model structures. Especially, by our proposed techniques, an earliest and simplest L2O model can be trained to outperform the latest complicated L2O models on a number of tasks. Our results demonstrate a greater potential of L2O yet to be unleashed, and urge to rethink the recent progress. Our codes are publicly available at: https://github.com/VITA-Group/L2O-Training-Techniques.