# Learning to optimize with convergence guarantees using nonlinear system theory

arXiv: https://arxiv.org/abs/2403.09389 · HTML: https://mlsanity.com/paper/2403.09389/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints](https://mlsanity.com/paper/1408.3595/) (2014)
- [Stable Recurrent Models](https://mlsanity.com/paper/1805.10369/) (2018)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [Learning to Optimize Neural Nets](https://mlsanity.com/paper/1703.00441/) (2017)
- [Safeguarded Learned Convex Optimization](https://mlsanity.com/paper/2003.01880/) (2020)
- [Neural System Level Synthesis: Learning over All Stabilizing Policies for Nonlinear Systems](https://mlsanity.com/paper/2203.11812/) (2022)
- [Optimization Algorithms as Robust Feedback Controllers](https://mlsanity.com/paper/2103.11329/) (2021)
- [Analysis and Design of First-Order Distributed Optimization Algorithms over Time-Varying Graphs](https://mlsanity.com/paper/1907.05448/) (2019)
- [Towards a Systems Theory of Algorithms](https://mlsanity.com/paper/2401.14029/) (2024)
- [The Analysis of Optimization Algorithms, A Dissipativity Approach](https://mlsanity.com/paper/2205.14264/) (2022)
- [Convex Synthesis of Accelerated Gradient Algorithms](https://mlsanity.com/paper/2102.06520/) (2021)
- [On Fundamental Proof Structures in First-Order Optimization](https://mlsanity.com/paper/2310.02015/) (2023)
- [Approximation by Simple Poles -- Part I: Density and Geometric Convergence Rate in Hardy Space](https://mlsanity.com/paper/2207.03982/) (2022)
- [Online Feedback Equilibrium Seeking](https://mlsanity.com/paper/2210.12088/) (2022)

## Built on by (most cited first, 7 of 7)

- [Learning to Boost the Performance of Stable Nonlinear Systems](https://mlsanity.com/paper/2405.00871/) (2024)
- [Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery](https://mlsanity.com/paper/2412.07544/) (2024)
- [Learning to optimize with guarantees: a complete characterization of linearly convergent algorithms](https://mlsanity.com/paper/2508.00775/) (2025)
- [From Learning to Optimize to Learning Optimization Algorithms](https://mlsanity.com/paper/2405.18222/) (2024)
- [Learning to accelerate Krasnosel'skii-Mann fixed-point iterations with guarantees](https://mlsanity.com/paper/2601.07665/) (2026)
- [MAD: A Magnitude And Direction Policy Parametrization for Stability Constrained Reinforcement Learning](https://mlsanity.com/paper/2504.02565/) (2025)
- [Learning Over-Relaxation Policies for ADMM with Convergence Guarantees](https://mlsanity.com/paper/2604.26932/) (2026)

## Abstract (arXiv)

The increasing reliance on numerical methods for controlling dynamical systems and training machine learning models underscores the need to devise algorithms that dependably and efficiently navigate complex optimization landscapes. Classical gradient descent methods offer strong theoretical guarantees for convex problems; however, they demand meticulous hyperparameter tuning for non-convex ones. The emerging paradigm of learning to optimize (L2O) automates the discovery of algorithms with optimized performance leveraging learning models and data - yet, it lacks a theoretical framework to analyze convergence of the learned algorithms. In this paper, we fill this gap by harnessing nonlinear system theory. Specifically, we propose an unconstrained parametrization of all convergent algorithms for smooth non-convex objective functions. Notably, our framework is directly compatible with automatic differentiation tools, ensuring convergence by design while learning to optimize.