mlsanityData to 14 Oct 2026

Predicting Large Model Test Losses with a Noisy Quadratic System

arXiv 2605.09154 ↗First posted 9 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce a predictive model that estimates the pre-training loss of large models from model size (N), batch size (B) and number of weight updates (K). This is the first loss prediction model that can handle changing batch size. The model outperforms Chinchilla's loss model, a model of the test loss using the batch size and number of tokens, in terms of projecting the loss at extrapolated compute budgets (up to 1000 folds). A natural use of the model is to find optimal N, B, K configurations under explicit and compound resource constraints like time, memory and compute. In our experiments, the model-selected configurations are close to ground-truth optimal. Our work advocates for loss prediction as a better alternative to heuristic-based laws, which are growing in complexity. The implementation is available on https://github.com/chuningxdy/Noisy-Quadratic-System.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

28 of 46 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  8. 2018An Empirical Model of Large-Batch Trainingcited by 281
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  10. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  11. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  12. 2022A Solvable Model of Neural Scaling Lawscited by 116
16 more
  1. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  2. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  3. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  4. 2024Chinchilla Scaling: A replication attemptcited by 72
  5. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  6. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  7. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  8. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  9. 2025Scaling Laws for Optimal Data Mixturescited by 52
  10. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  11. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  12. 2024Scaling Optimal LR Across Token Horizonscited by 40
  13. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 32
  14. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 25
  15. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  16. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22

Built on by

Not cited yet.