Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Not on the map. Why?
Abstract (arXiv)
Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 17 Feb 2026.
Builds on
48 of 63 references.
References · 48
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2025Qwen3 Technical Reportcited by 8,195
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023Mistral 7Bcited by 3,065
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
36 more
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2024Qwen2 Technical Reportcited by 2,076
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
- 2025Gemma 3 Technical Reportcited by 1,686
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2024Yi: Open Foundation Models by 01.AIcited by 706
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
- 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
- 2018An Empirical Model of Large-Batch Trainingcited by 268
- 2021Scaling Laws for Transfercited by 265
- 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
- 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 137
- 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
- 2024Language models scale reliably with over-training and on downstream taskscited by 105
- 2022Broken Neural Scaling Lawscited by 102
- 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
- 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
- 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
- 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
- 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
- 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
- 2024Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?cited by 39
- 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
- 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
- 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 27
- 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
- 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
- 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
- 2025EvoLM: In Search of Lost Language Model Training Dynamicscited by 16
- 2025Knowledge or Reasoning? A Close Look at How LLMs Think Across Domainscited by 14
- 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
- 2025Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based Perspectivecited by 7
- 2025Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Trainingcited by 6
- 2025Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learningcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
- 2024Revisiting the Superficial Alignment Hypothesis
- 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning
- 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead
- 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis
- 2026Brevity Constraints Reverse Performance Hierarchies in Language Models
- 2026Operationalising the Superficial Alignment Hypothesis via Task Complexity