# On the convergence of adaptive first order methods: proximal gradient and alternating minimization algorithms

arXiv: https://arxiv.org/abs/2311.18431 · HTML: https://mlsanity.com/paper/2311.18431/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 16 of 16)

- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes](https://mlsanity.com/paper/1805.08114/) (2018)
- [DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule](https://mlsanity.com/paper/2302.12022/) (2023)
- [Adaptive Gradient Descent without Descent](https://mlsanity.com/paper/1910.09529/) (2019)
- [Golden Ratio Algorithms for Variational Inequalities](https://mlsanity.com/paper/1803.08832/) (2018)
- [Adaptive Proximal Gradient Method for Convex Optimization](https://mlsanity.com/paper/2308.02261/) (2023)
- [Acceleration by Stepsize Hedging II: Silver Stepsize Schedule for Smooth Convex Optimization](https://mlsanity.com/paper/2309.16530/) (2023)
- [Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradient](https://mlsanity.com/paper/2301.04431/) (2023)
- [Adaptive Gradient Methods for Constrained Convex Optimization and Variational Inequalities](https://mlsanity.com/paper/2007.08840/) (2020)
- [A simple uniformly optimal method without line search for convex optimization](https://mlsanity.com/paper/2310.10082/) (2023)
- [Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method](https://mlsanity.com/paper/2206.06900/) (2022)
- [Accelerated Gradient Descent via Long Steps](https://mlsanity.com/paper/2309.09961/) (2023)
- [Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates](https://mlsanity.com/paper/2110.03274/) (2021)
- [Proximal Gradient Algorithms under Local Lipschitz Gradient Continuity: A Convergence and Robustness Analysis of PANOC](https://mlsanity.com/paper/2112.13000/) (2021)
- [The variable metric forward-backward splitting algorithm under mild differentiability assumptions](https://mlsanity.com/paper/1605.00952/) (2016)
- [On the convergence of proximal gradient methods for convex simple bilevel optimization](https://mlsanity.com/paper/2305.03559/) (2023)

## Built on by (most cited first, 2 of 2)

- [Adaptive proximal gradient methods are universal without approximation](https://mlsanity.com/paper/2402.06271/) (2024)
- [Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices](https://mlsanity.com/paper/2404.09617/) (2024)

## Abstract (arXiv)

Building upon recent works on linesearch-free adaptive proximal gradient methods, this paper proposes adaPGq,r, a framework that unifies and extends existing results by providing larger stepsize policies and improved lower bounds. Different choices of the parameters q and r are discussed and the efficacy of the resulting methods is demonstrated through numerical simulations. In an attempt to better understand the underlying theory, its convergence is established in a more general setting that allows for time-varying parameters. Finally, an adaptive alternating minimization algorithm is presented by exploring the dual setting. This algorithm not only incorporates additional adaptivity, but also expands its applicability beyond standard strongly convex settings.