mlsanityData to 7 Oct 2026

Learning Rate Schedules in the Presence of Distribution Shift

arXiv 2303.15634 ↗First posted 27 Mar 2023cs.LG math.OC stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We design learning rate schedules that minimize regret for SGD-based online learning in the presence of a changing data distribution. We fully characterize the optimal learning rate schedule for online linear regression via a novel analysis with stochastic differential equations. For general convex loss functions, we propose new learning rate schedules that are robust to distribution shift and we give upper and lower bounds for the regret that only differ by constants. For non-convex loss functions, we define a notion of regret based on the gradient norm of the estimated models and propose a learning schedule that minimizes an upper bound on the total expected regret. Intuitively, one expects changing loss landscapes to require more exploration, and we confirm that optimal learning rate schedules typically increase in the presence of distribution shift. Finally, we provide experiments for high-dimensional regression models and neural networks to illustrate these learning rate schedules and their cumulative regret.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

21 of 44 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  5. 2017How to Escape Saddle Points Efficientlycited by 284
  6. 2013Non-stationary Stochastic Optimizationcited by 173
  7. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  8. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  9. 2015Online Optimization : Competing with Dynamic Comparatorscited by 99
  10. 2014Deep learning with Elastic Averaging SGDcited by 90
  11. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  12. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
9 more
  1. 2016Tracking Slowly Moving Clairvoyant: Optimal Dynamic Regret of Online Learning with True and Noisy Gradientcited by 65
  2. 2018On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradientcited by 42
  3. 2018Averaging Stochastic Gradient Descent on Riemannian Manifoldscited by 39
  4. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  5. 2020On Learning Rates and Schr"odinger Operatorscited by 33
  6. 2017Deep Relaxation: partial differential equations for optimizing deep neural networkscited by 27
  7. 2017Tracking Moving Agents via Inexact Online Gradient Descent Algorithmcited by 6
  8. 2019Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometrycited by 6
  9. 2020How do SGD hyperparameters in natural training affect adversarial robustness?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 1; 2026: 1.

7 papers.

  1. 2023Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systemscited by 14
  2. 2023A Stability Principle for Learning under Non-Stationaritycited by 9
  3. 2024A call for embodied AIcited by 7
  4. 2023Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shiftcited by 4
  5. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 4
  6. 2024PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responsescited by 2
  7. 2025Efficient Hyperparameter Search for Non-Stationary Model Trainingcited by 0