mlsanityData to 14 Oct 2026

Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check

arXiv 2507.00885 ↗First posted 1 Jul 2025cs.CL cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple transformations of the performance metric, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, and we find that predictable scaling only occurs in a minority of cases: 39% of the time. Moreover, seemingly benign changes to the experimental setting can completely change the scaling behavior. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To accurately model the relationship between pretraining loss and task performance, we must embrace the cases in which scaling behavior deviates from linear trends.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 26 of its 44 references that are in the map. First posted on arXiv 1 Jul 2025.

Builds on

26 of 44 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2018CoQA: A Conversational Question Answering Challengecited by 683
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  11. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  12. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
14 more
  1. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  2. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  3. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  4. 2021Learning Curve Theorycited by 85
  5. 2022Training Trajectories of Language Models Across Scalescited by 81
  6. 2022Inverse scaling can become U-shapedcited by 76
  7. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 54
  8. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  9. 2024Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?cited by 40
  10. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  11. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  12. 2024Establishing Task Scaling Laws via Compute-Efficient Model Ladderscited by 35
  13. 2022Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experimentscited by 23
  14. 2022How Much More Data Do I Need? Estimating Requirements for Downstream Taskscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 24.

29 papers.

Sort:

By area:

  1. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  2. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  3. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  4. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
  5. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  6. 2025Relative Scaling Laws for LLMscited by 6
  7. 2025Data Scaling Laws for Radiology Foundation Modelscited by 5
  8. 2026TESSERA v2: Scaling Pixel-wise Earth Foundation Modelscited by 5
  9. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
  10. 2026Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergencecited by 3
  11. 2026Neural Neural Scaling Lawscited by 2
  12. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
17 more
  1. 2026Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimationcited by 2
  2. 2026Small-Scale Experiments: Are We There Yet?cited by 2
  3. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  4. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Taskscited by 1
  5. 2025What Scales in Cross-Entropy Scaling Law?cited by 1
  6. 2026The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuningcited by 1
  7. 2026How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?cited by 1
  8. 2026Prescriptive Scaling Reveals the Evolution of Language Model Capabilitiescited by 1
  9. 2026Capacity-Aware Mixture Law Enables Efficient LLM Data Optimizationcited by 1
  10. 2026Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floorcited by 1
  11. 2026A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agentscited by 1
  12. 2026Post-Training Science for Supervised Fine-Tuningcited by 1
  13. 2026Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AIcited by 0
  14. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0
  15. 2026Validity Threats for Foundation Model Researchcited by 0
  16. 2026When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learningcited by 0
  17. 2026Capability Scaling-Down Laws for LLM Compressioncited by 0