mlsanityData to 14 Oct 2026

Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit

arXiv 2410.05838 ↗First posted 8 Oct 2024cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

One of the main challenges in optimal scaling of large language models (LLMs) is the prohibitive cost of hyperparameter tuning, particularly learning rate η and batch size B. While techniques like μP (Yang et al., 2022) provide scaling rules for optimal η transfer in the infinite model size limit, the optimal scaling behavior in the infinite data size limit remains unknown. We fill in this gap by observing for the first time an intricate dependence of optimal η scaling on the pretraining token budget T, B and its relation to the critical batch size Bcrit, which we measure to evolve as Bcrit propto T. Furthermore, we show that the optimal batch size is positively correlated with Bcrit: keeping it fixed becomes suboptimal over time even if learning rate is scaled optimally. Surprisingly, our results demonstrate that the observed optimal η and B dynamics are preserved with μP model scaling, challenging the conventional view of Bcrit dependence solely on loss value. Complementing optimality, we examine the sensitivity of loss to changes in learning rate, where we find the sensitivity to decrease with increase of T and to remain constant with μP model scaling. We hope our results make the first step towards a unified picture of the joint optimal data and model scaling.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

34 of 43 references.

References · 34

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2016Layer Normalizationcited by 5,669
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  12. 2018An Empirical Model of Large-Batch Trainingcited by 281
22 more
  1. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  2. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  3. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  4. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  5. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  6. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  7. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  8. 2023A Spectral Condition for Feature Learningcited by 109
  9. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  10. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  11. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  12. 2024Scalable Optimization in the Modular Normcited by 68
  13. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  14. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 57
  15. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  16. 2021Batch size-invariance for policy optimizationcited by 35
  17. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  18. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 28
  19. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  20. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16
  21. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  22. 2023Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  2. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  3. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  4. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  5. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
  6. 2025Optimal Scaling Needs Optimal Normcited by 8
  7. 2025Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Trainingcited by 6
  8. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0
  9. 2026How to scale your HEP ML models: A recipe for robust architecture comparisons at scalecited by 0