Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
Not on the map. Why?
Abstract (arXiv)
Traditional scaling laws in natural language processing suggest that increasing model size and training data enhances performance. However, recent studies reveal deviations, particularly in large language models, where performance improvements decelerate, which is a phenomenon known as sub-scaling. This paper revisits these scaling laws by examining the impact of data quality and training strategies on model performance. Through extensive empirical analysis of over 400 models, we identify high data density and non-optimal resource allocation as key factors contributing to sub-scaling. High data density leads to diminishing returns due to redundant information, while optimal resource allocation is crucial for sustained performance improvements. We propose a sub-optimal scaling law that better predicts performance in sub-scaling regimes, highlighting the importance of data quality and diversity.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 13 Jul 2025.
Builds on
27 of 40 references.
References · 27
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
- 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2021Explaining Neural Scaling Lawscited by 353
15 more
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
- 2018An Empirical Model of Large-Batch Trainingcited by 268
- 2021Scaling Laws for Transfercited by 265
- 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
- 2021Scaling Laws for Neural Machine Translationcited by 110
- 2024How to Train Data-Efficient LLMscited by 91
- 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
- 2021Pareto Self-Supervised Training for Few-Shot Learningcited by 52
- 2019One Epoch Is All You Needcited by 41
- 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
- 2023Predicting Emergent Abilities with Infinite Resolution Evaluationcited by 28
- 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
- 2025On a Connection Between Imitation Learning and RLHFcited by 13
- 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scalingcited by 5
- 2024The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysiscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.