mlsanityData to 7 Oct 2026

Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints

arXiv 1408.3595 ↗First posted 15 Aug 2014math.OC cs.NA cs.SYcited by 105

Not on the map. Why?

Abstract (arXiv)

This manuscript develops a new framework to analyze and design iterative optimization algorithms built on the notion of Integral Quadratic Constraints (IQC) from robust control theory. IQCs provide sufficient conditions for the stability of complicated interconnected systems, and these conditions can be checked by semidefinite programming. We discuss how to adapt IQC theory to study optimization algorithms, proving new inequalities about convex functions and providing a version of IQC theory adapted for use by optimization researchers. Using these inequalities, we derive numerical upper bounds on convergence rates for the gradient method, the heavy-ball method, Nesterov's accelerated method, and related variants by solving small, simple semidefinite programming problems. We also briefly show how these techniques can be used to search for optimization algorithms with desired performance characteristics, establishing a new methodology for algorithm design.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Aug 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 17; 2023: 14; 2024: 15; 2025: 13; 2026: 21.

105 papers.

Sort:

By area:

  1. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  2. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  3. 2021Extragradient Method: O(1/K) Last-Iterate Convergence for Monotone Variational Inequalities and Connections With Cocoercivitycited by 39
  4. 2021Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimizationcited by 31
  5. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  6. 2022Tight Last-Iterate Convergence of the Extragradient and the Optimistic Gradient Descent-Ascent Algorithm for Constrained Monotone Variational Inequalitiescited by 18
  7. 2021Algorithmic Instabilities of Accelerated Gradient Descentcited by 16
  8. 2023On the Role of Server Momentum in Federated Learningcited by 16
  9. 2021Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systemscited by 15
  10. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  11. 2024On PI Controllers for Updating Lagrange Multipliers in Constrained Optimizationcited by 12
  12. 2022On the fast convergence of minibatch heavy ball momentumcited by 11
38 more
  1. 2022Provable Acceleration of Heavy Ball beyond Quadratics for a Class of Polyak-Lojasiewicz Functions when the Non-Convexity is Averaged-Outcited by 11
  2. 2022PEPit: computer-assisted worst-case analyses of first-order optimization methods in Pythoncited by 10
  3. 2022Nesterov Accelerated Shuffling Gradient Method for Convex Optimizationcited by 10
  4. 2024DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reductioncited by 10
  5. 2021Wasserstein distance estimates for the distributions of numerical approximations to ergodic stochastic differential equationscited by 8
  6. 2021Escaping Saddle Points Faster with Stochastic Momentumcited by 8
  7. 2021Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networkscited by 7
  8. 2021Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentumcited by 7
  9. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 7
  10. 2023Identifying Equivalent Training Dynamicscited by 7
  11. 2024Learning to optimize with convergence guarantees using nonlinear system theorycited by 7
  12. 2021Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimizationcited by 6
  13. 2021Linear systems with neural network nonlinearities: Improved stability analysis via acausal Zames-Falb multiplierscited by 6
  14. 2023Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descentcited by 6
  15. 2024Nesterov acceleration in benignly non-convex landscapescited by 6
  16. 2021Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problemscited by 5
  17. 2021Memory Augmented Optimizers for Deep Learningcited by 5
  18. 2024Data-Driven Performance Guarantees for Classical and Learned Optimizerscited by 5
  19. 2024Accelerated Preference Optimization for Large Language Model Alignmentcited by 5
  20. 2021Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimizationcited by 4
  21. 2022Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptoticscited by 4
  22. 2023A Nonstochastic Control Approach to Optimizationcited by 4
  23. 2024Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimizationcited by 4
  24. 2024Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networkscited by 4
  25. 2021Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditionerscited by 3
  26. 2022Convex Programs and Lyapunov Functions for Reinforcement Learning: A Unified Perspective on the Analysis of Value-Based Methodscited by 3
  27. 2022A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networkscited by 3
  28. 2023Diffusion Sampling with Momentum for Mitigating Divergence Artifactscited by 3
  29. 2024Distributed Event-Based Learning via ADMMcited by 3
  30. 2025Learning to optimize with guarantees: a complete characterization of linearly convergent algorithmscited by 3
  31. 2021Transient growth of accelerated optimization algorithmscited by 2
  32. 2021On Centralized and Distributed Mirror Descent: Convergence Analysis Using Quadratic Constraintscited by 2
  33. 2022Understanding A Class of Decentralized and Federated Optimization Algorithms: A Multi-Rate Feedback Control Perspectivecited by 2
  34. 2022Neural Observer with Lyapunov Stability Guarantee for Uncertain Nonlinear Systemscited by 2
  35. 2023The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Controlcited by 2
  36. 2023On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insightscited by 2
  37. 2023A Variational Perspective on High-Resolution ODEscited by 2
  38. 2023Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence Beyond the Minty Propertycited by 2

50 most cited of 105 All 105 (JSON)