mlsanityData to 7 Oct 2026

How predictable is language model benchmark performance?

arXiv 2401.04757 ↗First posted 9 Jan 2024cs.LG cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

We investigate large language model performance across five orders of magnitude of compute scaling in eleven recent model architectures. We show that average benchmark performance, aggregating over many individual tasks and evaluations as in the commonly-used BIG-Bench dataset, is decently predictable as a function of training compute scale. Specifically, when extrapolating BIG-Bench Hard performance across one order of magnitude in compute, we observe average absolute errors of 6 percentage points (pp). By contrast, extrapolation for individual BIG-Bench tasks across an order of magnitude in compute yields higher average errors of 18pp. Nonetheless, individual task performance remains significantly more predictable than chance. Overall, our work suggests compute scaling provides a promising basis to forecast AI capabilities in diverse benchmarks, though predicting performance in specific tasks poses challenges.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

24 of 35 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
12 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023BloombergGPT: A Large Language Model for Financecited by 923
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  8. 2022Predictability and Surprise in Large Generative Modelscited by 209
  9. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  10. 2022Broken Neural Scaling Lawscited by 102
  11. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  12. 2022Algorithmic progress in computer visioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 19; 2026: 7.

36 papers.

Sort:

By area:

  1. 2025Humanity's Last Examcited by 467
  2. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  3. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  4. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  5. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  6. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  7. 2025A Rosetta Stone for AI Benchmarkscited by 29
  8. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 27
  9. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  10. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24
  11. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  12. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 23
24 more
  1. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  2. 2024Predicting Emergent Capabilities by Finetuningcited by 10
  3. 2024Collaborative Performance Prediction for Large Language Modelscited by 9
  4. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 8
  5. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  6. 2025Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspectivecited by 7
  7. 2024Performance Law of Large Language Modelscited by 6
  8. 2025Forecasting Frontier Language Model Agent Capabilitiescited by 5
  9. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  10. 2025Synthetic bootstrapped pretrainingcited by 4
  11. 2025Inference Compute-Optimal Video Vision Language Modelscited by 3
  12. 2025The Price of Progress: Price Performance and the Future of AIcited by 3
  13. 2025Meek Models Shall Inherit the Earthcited by 2
  14. 2025Predicting Task Performance with Context-aware Scaling Lawscited by 2
  15. 2025A Latent Variable Framework for Scaling Laws in Large Language Modelscited by 2
  16. 2026Classroom Final Exam: An Instructor-Tested Reasoning Benchmarkcited by 2
  17. 2026Leaderboard Incentives: Model Rankings under Strategic Post-Trainingcited by 2
  18. 2025Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Performancecited by 1
  19. 2025Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Lawcited by 1
  20. 2026SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scalingcited by 1
  21. 2026Post-Training Science for Supervised Fine-Tuningcited by 1
  22. 2026Continually self-improving AIcited by 0
  23. 2026Validity Threats for Foundation Model Researchcited by 0
  24. 2026Two AI Metrics Diverged: Will it Make All the Difference?cited by 0