mlsanityData to 14 Oct 2026

Observational Scaling Laws and the Predictability of Language Model Performance

arXiv 2405.10938 ↗First posted 17 May 2024cs.LG cs.AI cs.CLcited by 130

Not on the map. Why?

Abstract (arXiv)

Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach to building this understanding, but the requirement of training models across many different scales has limited their use. We propose an alternative, observational approach that bypasses model training and instead builds scaling laws from ~100 publically available models. Building a single scaling law from multiple model families is challenging due to large variations in their training compute efficiencies and capabilities. However, we show that these variations are consistent with a simple, generalized scaling law where language model performance is a function of a low-dimensional capability space, and model families only vary in their efficiency in converting training compute to capabilities. Using this approach, we show the surprising predictability of complex scaling phenomena: we show that several emergent phenomena follow a smooth, sigmoidal behavior and are predictable from small models; we show that the agent performance of models such as GPT-4 can be precisely predicted from simpler non-agentic benchmarks; and we show how to predict the impact of post-training interventions like Chain-of-Thought and Self-Consistency as language model capabilities continue to improve.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Builds on 54 of its 101 references that are in the map. First posted on arXiv 17 May 2024.

Builds on

54 of 101 references.

References · 54

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023Mistral 7Bcited by 3,164
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2024Mixtral of Expertscited by 1,823
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2023StarCoder: may the source be with you!cited by 799
  12. 2024Yi: Open Foundation Models by 01.AIcited by 719
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  14. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  15. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  16. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  17. 2023The Falcon Series of Open Language Modelscited by 535
  18. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  19. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  20. 2021Explaining Neural Scaling Lawscited by 363
  21. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  22. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  23. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  24. 2021Scaling Laws for Transfercited by 272
  25. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  26. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  27. 2022Holistic Evaluation of Language Modelscited by 131
  28. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  29. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  30. 2021Scaling Laws for Neural Machine Translationcited by 111
  31. 2022Broken Neural Scaling Lawscited by 109
  32. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  33. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  34. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  35. 2022Training Trajectories of Language Models Across Scalescited by 81
  36. 2023Efficient Benchmarking of Language Modelscited by 71
  37. 2024Compression Represents Intelligence Linearlycited by 56
  38. 2023Revealing the structure of language model capabilitiescited by 40

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 55; 2026: 45.

130 papers.

Sort:

By area:

  1. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  2. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  3. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  4. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  5. 2024Scaling Laws for Precisioncited by 95
  6. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 72
  7. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  8. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 59
  9. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  10. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  11. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  12. 2024Memory3: Language Modeling with Explicit Memorycited by 40
38 more
  1. 2025Video-T1: Test-Time Scaling for Video Generationcited by 40
  2. 2025A Rosetta Stone for AI Benchmarkscited by 36
  3. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  4. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  5. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  6. 2024Evaluating Language Models as Synthetic Data Generatorscited by 33
  7. 2024Taking AI Welfare Seriouslycited by 31
  8. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 31
  9. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  10. 2025Pre-training under infinite computecited by 29
  11. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  12. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  13. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 26
  14. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 26
  15. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24
  16. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 24
  17. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  18. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 18
  19. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  20. 2025Scaling Inference-Efficient Language Modelscited by 16
  21. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  22. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  23. 2024DevBench: A multimodal developmental benchmark for language learningcited by 13
  24. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 13
  25. 2025NegVQA: Can Vision Language Models Understand Negation?cited by 13
  26. 2025Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluationcited by 12
  27. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 12
  28. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  29. 2024Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Optionscited by 11
  30. 2024Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LMcited by 11
  31. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 10
  32. 2024Predicting Emergent Capabilities by Finetuningcited by 10
  33. 2026Scaling Self-Play with Self-Guidancecited by 10
  34. 2024Collaborative Performance Prediction for Large Language Modelscited by 9
  35. 2024Statistical inference on black-box generative models in the data kernel perspective spacecited by 9
  36. 2025Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Frameworkcited by 9
  37. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Modelscited by 9
  38. 2025A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Managementcited by 8

50 most cited of 130 All 130 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SLDBench v2: Active Scaling Law Discovery
  2. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks
  3. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  4. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  5. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  6. 2025Emergent Abilities in Large Language Models: A Survey
  7. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
  8. 2025Can Language Models Discover Scaling Laws?