# Learning to Warm-Start Fixed-Point Optimization Algorithms

arXiv: https://arxiv.org/abs/2309.07835 · HTML: https://mlsanity.com/paper/2309.07835/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 123 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 30 of 63)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [The Case for Learned Index Structures](https://mlsanity.com/paper/1712.01208/) (2017)
- [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017)
- [Algorithms with Predictions](https://mlsanity.com/paper/2006.09123/) (2020)
- [Global Convergence of ADMM in Nonconvex Nonsmooth Optimization](https://mlsanity.com/paper/1511.06324/) (2015)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Learning Optimal Solutions for Extremely Fast AC Optimal Power Flow](https://mlsanity.com/paper/1910.01213/) (2019)
- [Near-Optimal Bounds for Online Caching with Machine Learned Advice](https://mlsanity.com/paper/1910.12172/) (2019)
- [One Network to Solve Them All --- Solving Linear Inverse Problems using Deep Projection Models](https://mlsanity.com/paper/1703.09912/) (2017)
- [Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems](https://mlsanity.com/paper/1410.1390/) (2014)
- [A PAC Approach to Application-Specific Algorithm Selection](https://mlsanity.com/paper/1511.07147/) (2015)
- [Efficient representation and approximation of model predictive control laws via deep learning](https://mlsanity.com/paper/1806.10644/) (2018)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Faster Matchings via Learned Duals](https://mlsanity.com/paper/2107.09770/) (2021)
- [Deep Unfolding for Communications Systems: A Survey and Some New Directions](https://mlsanity.com/paper/1906.05774/) (2019)
- [Acceleration Methods](https://mlsanity.com/paper/2101.09545/) (2021)
- [Unrolled Optimization with Deep Priors](https://mlsanity.com/paper/1705.08041/) (2017)
- [Online Mixed-Integer Optimization in Milliseconds](https://mlsanity.com/paper/1907.02206/) (2019)

## Built on by (most cited first, 30 of 32)

- [FSNet: Feasibility-Seeking Neural Network for Constrained Optimization with Guarantees](https://mlsanity.com/paper/2506.00362/) (2025)
- [Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers](https://mlsanity.com/paper/2508.10480/) (2025)
- [Differentiation Through Black-Box Quadratic Programming Solvers](https://mlsanity.com/paper/2410.06324/) (2024)
- [Aligning Diffusion Model with Problem Constraints for Trajectory Optimization](https://mlsanity.com/paper/2504.00342/) (2025)
- [Data-Driven Performance Guarantees for Classical and Learned Optimizers](https://mlsanity.com/paper/2404.13831/) (2024)
- [Learning Multiple Initial Solutions to Optimization Problems](https://mlsanity.com/paper/2411.02158/) (2024)
- [A First-order Generative Bilevel Optimization Framework for Diffusion Models](https://mlsanity.com/paper/2502.08808/) (2025)
- [Deep FlexQP: Accelerated Nonlinear Programming via Deep Unfolding](https://mlsanity.com/paper/2512.01565/) (2025)
- [Learning to optimize with guarantees: a complete characterization of linearly convergent algorithms](https://mlsanity.com/paper/2508.00775/) (2025)
- [Learning to accelerate distributed ADMM using graph neural networks](https://mlsanity.com/paper/2509.05288/) (2025)
- [From Learning to Optimize to Learning Optimization Algorithms](https://mlsanity.com/paper/2405.18222/) (2024)
- [WARP-LCA: Efficient Convolutional Sparse Coding with Locally Competitive Algorithm](https://mlsanity.com/paper/2410.18794/) (2024)
- [Learning to accelerate Krasnosel'skii-Mann fixed-point iterations with guarantees](https://mlsanity.com/paper/2601.07665/) (2026)
- [Scalable Deep Unfolding of Conic Optimizers](https://mlsanity.com/paper/2606.13825/) (2026)
- [Robotic warehousing operations: a learn-then-optimize approach to large-scale neighborhood search](https://mlsanity.com/paper/2408.16890/) (2024)
- [Cheap Thrills: Effective Amortized Optimization Using Inexpensive Labels](https://mlsanity.com/paper/2603.05495/) (2026)
- [Parametric Nonconvex Optimization via Convex Surrogates](https://mlsanity.com/paper/2604.05640/) (2026)
- [GLENS: Global Search via Learning from Solver Iterates with Diffusion Models](https://mlsanity.com/paper/2606.00366/) (2026)
- [LEAF: A Learning-Enabled ADMM Framework for Accelerated Convex Optimization](https://mlsanity.com/paper/2606.08993/) (2026)
- [Large Language Models for Knowledge-Free Network Management: Feasibility Study and Opportunities](https://mlsanity.com/paper/2410.17259/) (2024)
- [Preserving Expert-Level Privacy in Offline Reinforcement Learning](https://mlsanity.com/paper/2411.13598/) (2024)
- [Learning to Solve Constrained Bilevel Control Co-Design Problems](https://mlsanity.com/paper/2507.09050/) (2025)
- [Reinforcement learning for adaptive interior point methods in convex quadratic programming](https://mlsanity.com/paper/2509.07404/) (2025)
- [Flow-Opt: Scalable Centralized Multi-Robot Trajectory Optimization with Flow Matching and Differentiable Optimization](https://mlsanity.com/paper/2510.09204/) (2025)
- [KANFormer for Predicting Fill Probabilities via Survival Analysis in Limit Order Books](https://mlsanity.com/paper/2512.05734/) (2025)
- [partialCBDs: Differentiable Causal Block Diagrams](https://mlsanity.com/paper/2602.07581/) (2026)
- [Learning Over-Relaxation Policies for ADMM with Convergence Guarantees](https://mlsanity.com/paper/2604.26932/) (2026)
- [WARP: A Benchmark for Primal-Dual Warm-Starting of Interior-Point Solvers](https://mlsanity.com/paper/2605.05728/) (2026)
- [Distribution-Aware Algorithm Design with LLM Agents](https://mlsanity.com/paper/2605.14141/) (2026)
- [Memory-Computation Tradeoffs in Semi Amortized Parametric Optimization](https://mlsanity.com/paper/2607.20769/) (2026)

## Abstract (arXiv)

We introduce a machine-learning framework to warm-start fixed-point optimization algorithms. Our architecture consists of a neural network mapping problem parameters to warm starts, followed by a predefined number of fixed-point iterations. We propose two loss functions designed to either minimize the fixed-point residual or the distance to a ground truth solution. In this way, the neural network predicts warm starts with the end-to-end goal of minimizing the downstream loss. An important feature of our architecture is its flexibility, in that it can predict a warm start for fixed-point algorithms run for any number of steps, without being limited to the number of steps it has been trained on. We provide PAC-Bayes generalization bounds on unseen data for common classes of fixed-point operators: contractive, linearly convergent, and averaged. Applying this framework to well-known applications in control, statistics, and signal processing, we observe a significant reduction in the number of iterations and solution time required to solve these problems, through learned warm starts.