# Super-Acceleration with Cyclical Step-sizes

arXiv: https://arxiv.org/abs/2106.09687 · HTML: https://mlsanity.com/paper/2106.09687/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2021.

## Built on by (most cited first, 14 of 14)

- [Why Transformers Need Adam: A Hessian Perspective](https://mlsanity.com/paper/2402.16788/) (2024)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Provably Faster Gradient Descent via Long Steps](https://mlsanity.com/paper/2307.06324/) (2023)
- [From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression](https://mlsanity.com/paper/2310.01687/) (2023)
- [FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity](https://mlsanity.com/paper/2404.09816/) (2024)
- [Negative Stepsizes Make Gradient-Descent-Ascent Converge](https://mlsanity.com/paper/2505.01423/) (2025)
- [Fast-Convergent Federated Learning via Cyclic Aggregation](https://mlsanity.com/paper/2210.16520/) (2022)
- [Towards a Better Theoretical Understanding of Independent Subnetwork Training](https://mlsanity.com/paper/2306.16484/) (2023)
- [Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates](https://mlsanity.com/paper/2206.00832/) (2022)
- [Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate](https://mlsanity.com/paper/2401.03058/) (2024)
- [Polynomial Preconditioning for Gradient Methods](https://mlsanity.com/paper/2301.13194/) (2023)
- [When is Momentum Extragradient Optimal? A Polynomial-Based Analysis](https://mlsanity.com/paper/2211.04659/) (2022)
- [On the Error-Propagation of Inexact Hotelling's Deflation for Principal Component Analysis](https://mlsanity.com/paper/2310.04283/) (2023)
- [Learning Rate Annealing Improves Tuning Robustness in Stochastic Optimization](https://mlsanity.com/paper/2503.09411/) (2025)

## Abstract (arXiv)

We develop a convergence-rate analysis of momentum with cyclical step-sizes. We show that under some assumption on the spectral gap of Hessians in machine learning, cyclical step-sizes are provably faster than constant step-sizes. More precisely, we develop a convergence rate analysis for quadratic objectives that provides optimal parameters and shows that cyclical learning rates can improve upon traditional lower complexity bounds. We further propose a systematic approach to design optimal first order methods for quadratic minimization with a given spectral structure. Finally, we provide a local convergence rate analysis beyond quadratic minimization for the proposed methods and illustrate our findings through benchmarks on least squares and logistic regression problems.