mlsanityData to 14 Oct 2026

(Mis)Fitting: A Survey of Scaling Laws

arXiv 2502.18969 ↗First posted 26 Feb 2025cs.LG cs.AI cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Modern foundation models rely heavily on using scaling laws to guide crucial training decisions. Researchers often extrapolate the optimal architecture and hyper parameters settings from smaller training runs by describing the relationship between, loss, or task performance, and scale. All components of this process vary, from the specific equation being fit, to the training setup, to the optimization method. Each of these factors may affect the fitted law, and therefore, the conclusions of a given study. We discuss discrepancies in the conclusions that several prior works reach, on questions such as the optimal token to parameter ratio. We augment this discussion with our own analysis of the critical impact that changes in specific details may effect in a scaling study, and the resulting altered conclusions. Additionally, we survey over 50 papers that study scaling trends: while 45 of these papers quantify these trends using a power law, most under-report crucial details needed to reproduce their findings. To mitigate this, we we propose a checklist for authors to consider while contributing to scaling law research.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 69 of its 80 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

69 of 80 references.

References · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2021Scaling Vision Transformerscited by 999
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  13. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  14. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  15. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  16. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  17. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  18. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  19. 2023Scaling Data-Constrained Language Modelscited by 403
  20. 2021Explaining Neural Scaling Lawscited by 363
  21. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  22. 2018An Empirical Model of Large-Batch Trainingcited by 281
  23. 2021Scaling Laws for Transfercited by 272
  24. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  25. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  26. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  27. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  28. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  29. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  30. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  31. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  32. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  33. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  34. 2021Scaling Laws for Neural Machine Translationcited by 111
  35. 2022Broken Neural Scaling Lawscited by 109
  36. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  37. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  38. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 12.

26 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  3. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  4. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  5. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  6. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  7. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  8. 2025Literature Review Of Multi-Agent Debate For Problem-Solvingcited by 15
  9. 2026Scaling Self-Play with Self-Guidancecited by 10
  10. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 9
  11. 2025Scaling Laws are Redundancy Lawscited by 3
  12. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
14 more
  1. 2026Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selectioncited by 3
  2. 2025Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networkscited by 2
  3. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  4. 2026Slicing and Dicing: Configuring Optimal Mixtures of Expertscited by 2
  5. 2025LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thoughtcited by 1
  6. 2026WiCompass: Oracle-driven Data Scaling for mmWave Human Pose Estimationcited by 1
  7. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  8. 2026Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Losscited by 1
  9. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraintscited by 0
  10. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  11. 2026Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolationcited by 0
  12. 2026Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained Worldcited by 0
  13. 2026ScAn-Bench: Evaluating Scaling Analysis Methodologycited by 0
  14. 2026SLDBench v2: Active Scaling Law Discoverycited by 0