mlsanityData to 14 Oct 2026

Improved Scaling Laws in Linear Regression via Data Reuse

arXiv 2506.08415 ↗First posted 10 Jun 2025cs.LG math.ST stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws suggest that the test error of large language models trained online decreases polynomially as the model size and data size increase. However, such scaling can be unsustainable when running out of new data. In this work, we show that data reuse can improve existing scaling laws in linear regression. Specifically, we derive sharp test error bounds on M-dimensional linear models trained by multi-pass stochastic gradient descent (multi-pass SGD) on N data with sketched features. Assuming that the data covariance has a power-law spectrum of degree a, and that the true parameter follows a prior with an aligned power-law spectrum of degree b-a (with a > b > 1), we show that multi-pass SGD achieves a test error of Theta(M1-b + L(1-b)/a), where L lesssim Na/b is the number of iterations. In the same setting, one-pass SGD only attains a test error of Theta(M1-b + N(1-b)/a) (see e.g., Lin et al., 2024). This suggests an improved scaling law via data reuse (i.e., choosing L>N) in data-constrained regimes. Numerical simulations are also provided to verify our theoretical findings.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

37 of 51 references.

References · 37

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2021Scaling Vision Transformerscited by 999
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  5. 2019Benign Overfitting in Linear Regressioncited by 549
  6. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  7. 2023Scaling Data-Constrained Language Modelscited by 403
  8. 2021Explaining Neural Scaling Lawscited by 363
  9. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  10. 2023The Quantization Model of Neural Scalingcited by 138
  11. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  12. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
25 more
  1. 2022A Solvable Model of Neural Scaling Lawscited by 116
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  3. 2022Broken Neural Scaling Lawscited by 109
  4. 2024Scaling Laws for Precisioncited by 95
  5. 2021Learning Curve Theorycited by 85
  6. 2024Chinchilla Scaling: A replication attemptcited by 72
  7. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  8. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  9. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  10. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  11. 2025Distillation Scaling Lawscited by 67
  12. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  13. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  14. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  15. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 52
  16. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  17. 2021Last iterate convergence of SGD for Least-Squares in the Interpolation regimecited by 42
  18. 2020Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Modelcited by 39
  19. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 38
  20. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  21. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 32
  22. 2017A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)cited by 26
  23. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 26
  24. 2019Beating SGD Saturation with Tail-Averaging and Minibatchingcited by 21
  25. 2022Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regimecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  2. 2026High-Dimensional Statistics: Reflections on Progress and Open Problemscited by 4
  3. 2026Predicting and improving test-time scaling laws via reward tail-guided searchcited by 2
  4. 2026Asymmetric Scaling Laws from Sparse Featurescited by 1
  5. 2026Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biasescited by 0
  6. 2026Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scalingcited by 0
  7. 2026The Fourth Quadrant: A Stylized View of Benign Misfittingcited by 0
  8. 2026SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Lawscited by 0
  9. 2026Learning with Synthetic Data via SGD in High-Dimensional Linear Regressioncited by 0
  10. 2026Minibatch persistency, eight years later: what batch reuse costs in steps and joules, and what it saves in datacited by 0
  11. 2026Linear RNN Scaling Laws: When Longer Sequences Beat More Sequencescited by 0