mlsanityData to 7 Oct 2026

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

arXiv 2401.00448 ↗First posted 31 Dec 2023cs.LG cs.CLcited by 155

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data. However, these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference. We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand. We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal. Furthermore, we train 47 models of varying sizes and parameter counts to validate our formula and find that model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000). Finally, we ablate the procedure used to fit the Chinchilla scaling law coefficients and find that developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

32 of 61 references.

References · 32

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
20 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  7. 2018CoQA: A Conversational Question Answering Challengecited by 674
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  13. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  14. 2021Scaling Laws for Transfercited by 265
  15. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  16. 2023The Quantization Model of Neural Scalingcited by 137
  17. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  18. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  19. 2022Broken Neural Scaling Lawscited by 102
  20. 2019Beyond Human-Level Accuracy: Computational Challenges in Deep Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 63; 2026: 49.

155 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  4. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  5. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  6. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  7. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  8. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  9. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  10. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  11. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  12. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 102
38 more
  1. 2024Scaling Laws for Precisioncited by 92
  2. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  3. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  4. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  5. 2024Chinchilla Scaling: A replication attemptcited by 65
  6. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  7. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 57
  8. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
  9. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  10. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  11. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  12. 2024Densing Law of LLMscited by 39
  13. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  14. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  15. 2024Scaling Optimal LR Across Token Horizonscited by 35
  16. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  17. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 30
  18. 2024Not All LLM Reasoners Are Created Equalcited by 29
  19. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  20. 2025Parallel Scaling Law for Language Modelscited by 29
  21. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  22. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  23. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  24. 2025Pre-training under infinite computecited by 24
  25. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  26. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  27. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 23
  28. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  29. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 20
  30. 2024Toward Inference-optimal Mixture-of-Expert Large Language Modelscited by 19
  31. 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
  32. 2025Scaling Laws of Motion Forecasting and Planning -- Technical Reportcited by 19
  33. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  34. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 18
  35. 2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
  36. 2024How Much Can We Forget about Data Contamination?cited by 14
  37. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  38. 2025Scaling Inference-Efficient Language Modelscited by 14

50 most cited of 155 All 155 (JSON)