mlsanityData to 14 Oct 2026

Scaling Law for Language Models Training Considering Batch Size

arXiv 2412.01505 ↗First posted 2 Dec 2024cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have made remarkable advances in recent years, with scaling laws playing a critical role in this rapid progress. In this paper, we empirically investigate how a critical hyper-parameter, i.e., the global batch size, influences the LLM training prdocess. We begin by training language models ranging from 125 million to 2.6 billion parameters, using up to 300 billion high-quality tokens. Through these experiments, we establish a basic scaling law on model size and training data amount. We then examine how varying batch sizes and learning rates affect the convergence and generalization of these models. Our analysis yields batch size scaling laws under two different cases: with a fixed compute budget, and with a fixed amount of training data. Extrapolation experiments on models of increasing sizes validate our predicted laws, which provides guidance for optimizing LLM training strategies under specific resource constraints.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

19 of 25 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  11. 2018An Empirical Model of Large-Batch Trainingcited by 281
  12. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
7 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  2. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  3. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  4. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  5. 2024Scaling Law with Learning Rate Annealingcited by 29
  6. 2018SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learningcited by 25
  7. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 11.

22 papers.

Sort:

By area:

  1. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  2. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
  3. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 11
  4. 2025Quantum Variational Activation Functions Empower Kolmogorov-Arnold Networkscited by 10
  5. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  6. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  7. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  8. 2025NoLoCo: No-all-reduce Low Communication Training Method for Large Modelscited by 5
  9. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  10. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  11. 2025Scaling Laws for Code: A More Data-Hungry Regimecited by 3
  12. 2025MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Datacited by 2
10 more
  1. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  2. 2026Scaling Laws for Dynamic Mini-Batch SGD in Sketched Linear Regressioncited by 2
  3. 2025SOLAR: Scalable Optimization of Large-scale Architecture for Reasoningcited by 1
  4. 2026Towards joint scaling laws with optimal batch size schedulescited by 1
  5. 2026Scaling Inherently Interpretable Language Modelscited by 1
  6. 2026How to Set the Batch Size for Large-Scale Pre-training?cited by 0
  7. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  8. 2026ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Trainingcited by 0
  9. 2026Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-trainingcited by 0
  10. 2026Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffscited by 0