mlsanityData to 14 Oct 2026

Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

arXiv 2610.16518 ↗First posted 12 Oct 2026cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across eigendirections. Under power-law source and capacity conditions, we derive the exact finite-time risk and asymptotic equivalents of constant and Warmup-Stable-Decay (WSD) schedules. With a tuned base learning-rate, the resulting scaling law decays as Theta(T-q), where T is the horizon, and q is independent of the cooldown length c ∈ (0,1] but is strictly smaller for the constant schedule: any cooldown of fixed relative length c > 0, however short, improves the risk by a polynomial factor, while its length only affects the constant, which is minimized by linear decay (c = 1). Finally, we show that the optimal risk depends on the batch size b and horizon T only through bT, up to a critical batch size.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

13 of 20 references.

References · 13

  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  2. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  3. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  4. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  5. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  6. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 42
  7. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  8. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  9. 2025Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescited by 17
  10. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  11. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  12. 2026Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcited by 7
1 more
  1. 2025On the Effectiveness of the z-Transform Method in Quadratic Optimizationcited by 3

Built on by

Not cited yet.