# Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules

arXiv: https://arxiv.org/abs/2509.19189 · HTML: https://mlsanity.com/paper/2509.19189/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2025.

## Built on by (most cited first, 17 of 17)

- [Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging](https://mlsanity.com/paper/2602.03702/) (2026)
- [Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay](https://mlsanity.com/paper/2602.06797/) (2026)
- [Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws](https://mlsanity.com/paper/2602.14208/) (2026)
- [Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining](https://mlsanity.com/paper/2608.24814/) (2026)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [Scaling Domain Data Repetition in LLM Pretraining](https://mlsanity.com/paper/2608.14071/) (2026)
- [Effective Frontiers: A Unification of Neural Scaling Laws](https://mlsanity.com/paper/2602.02593/) (2026)
- [Asymmetric Scaling Laws from Sparse Features](https://mlsanity.com/paper/2605.23591/) (2026)
- [Explaining Data Mixing Scaling Laws](https://mlsanity.com/paper/2606.08167/) (2026)
- [Smooth Scaling Laws Hide Stepwise Token Learning](https://mlsanity.com/paper/2606.29858/) (2026)
- [When Does Learning Renormalize? Sufficient Conditions for Power Law Spectral Dynamics](https://mlsanity.com/paper/2512.18209/) (2025)
- [How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws](https://mlsanity.com/paper/2605.25698/) (2026)
- [Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining](https://mlsanity.com/paper/2608.28442/) (2026)
- [Learning with Synthetic Data via SGD in High-Dimensional Linear Regression](https://mlsanity.com/paper/2609.09572/) (2026)
- [Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent](https://mlsanity.com/paper/2609.16262/) (2026)
- [Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization](https://mlsanity.com/paper/2610.05329/) (2026)
- [Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features](https://mlsanity.com/paper/2610.16518/) (2026)

## Abstract (arXiv)

Scaling laws have emerged as a unifying lens for understanding and guiding the training of large language models (LLMs). However, existing studies predominantly focus on the final-step loss, leaving open whether the entire loss dynamics obey similar laws and, crucially, how the learning rate schedule (LRS) shapes them. We address these gaps in a controlled theoretical setting by analyzing stochastic gradient descent (SGD) on a power-law kernel regression model. The key insight is a novel intrinsic-time viewpoint, which captures the training progress more faithfully than iteration count. We then establish a Functional Scaling Law (FSL) that captures the full loss trajectory under arbitrary LRSs, with the schedule's influence entering through a simple convolutional functional. We further instantiate the theory for three representative LRSs -- constant, exponential decay, and warmup-stable-decay (WSD) -- and derive explicit scaling relations in both data- and compute-limited regimes. These comparisons explain key empirical phenomena: (i) higher-capacity models are more data- and compute-efficient; (ii) learning-rate decay improves training efficiency; and (iii) WSD-type schedules outperform pure decay. Finally, experiments on LLMs ranging from 0.1B to 1B parameters demonstrate the practical relevance of FSL as a surrogate model for fitting and predicting loss trajectories in large-scale pre-training.