# From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

arXiv: https://arxiv.org/abs/2609.40148 · HTML: https://mlsanity.com/paper/2609.40148/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator](https://mlsanity.com/paper/2403.08938/) (2024)
- [Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling](https://mlsanity.com/paper/2409.15156/) (2024)
- [On the origin of neural scaling laws: from random graphs to natural language](https://mlsanity.com/paper/2601.10684/) (2026)
- [A Defense of the Quadratic Model](https://mlsanity.com/paper/2607.21716/) (2026)
- [Universal One-third Time Scaling in Learning Peaked Distributions](https://mlsanity.com/paper/2602.03685/) (2026)
- [Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model](https://mlsanity.com/paper/2602.04774/) (2026)
- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Training Neural Networks at Any Scale](https://mlsanity.com/paper/2511.11163/) (2025)
- [Towards joint scaling laws with optimal batch size schedules](https://mlsanity.com/paper/2607.27731/) (2026)

## Abstract (arXiv)

Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features. Conditional on the representation, an exact Volterra equation separates two response components: a forcing term that propagates unresolved target error and a memory kernel that propagates stochastic-error injections. We prove that either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling; individual eigenvalues and target coefficients need not obey coordinatewise power laws. Under a joint schedule, intrinsic time Tt=sums<tηs controls optimization progress, while rt=Bt/ηt controls noise injection. Their interaction yields sharp conditions under which a schedule preserves, changes, or destroys the clean power law, together with a memory ceiling on noise reduction. The power-law random-feature model realizes this mechanism in 3+3(+2) propagation regimes with phase-dependent compute rates. Controlled nanoGPT experiments show that (1) learning-rate and batch-size schedules with matched B/η paths are nearly equivalent in intrinsic time, (2) a forcing-memory surrogate accurately predicts loss across schedules, and (3) its fitted exponents across real-world datasets identify the regime of LLMs in 3+3(+2) map. The source code for reproducing our experiments is available on https://github.com/yichenblue/spectra-to-schedules-in-pretraining.