# Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

arXiv: https://arxiv.org/abs/2610.16518 · HTML: https://mlsanity.com/paper/2610.16518/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations](https://mlsanity.com/paper/2405.18392/) (2024)
- [The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares](https://mlsanity.com/paper/1904.12838/) (2019)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [Optimal Linear Decay Learning Rate Schedules and Further Refinements](https://mlsanity.com/paper/2310.07831/) (2023)
- [Last iterate convergence of SGD for Least-Squares in the Interpolation regime](https://mlsanity.com/paper/2102.03183/) (2021)
- [The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training](https://mlsanity.com/paper/2501.18965/) (2025)
- [Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law](https://mlsanity.com/paper/2505.19227/) (2025)
- [Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules](https://mlsanity.com/paper/2509.19189/) (2025)
- [Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging](https://mlsanity.com/paper/2602.03702/) (2026)
- [Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay](https://mlsanity.com/paper/2602.06797/) (2026)
- [Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model](https://mlsanity.com/paper/2602.04774/) (2026)
- [On the Effectiveness of the z-Transform Method in Quadratic Optimization](https://mlsanity.com/paper/2507.03404/) (2025)

## Abstract (arXiv)

We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across eigendirections. Under power-law source and capacity conditions, we derive the exact finite-time risk and asymptotic equivalents of constant and Warmup-Stable-Decay (WSD) schedules. With a tuned base learning-rate, the resulting scaling law decays as Theta(T-q), where T is the horizon, and q is independent of the cooldown length c ∈ (0,1] but is strictly smaller for the constant schedule: any cooldown of fixed relative length c > 0, however short, improves the risk by a polynomial factor, while its length only affects the constant, which is minimized by linear decay (c = 1). Finally, we show that the optimal risk depends on the batch size b and horizon T only through bT, up to a critical batch size.