# Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check

arXiv: https://arxiv.org/abs/2507.00885 · HTML: https://mlsanity.com/paper/2507.00885/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Language models scale reliably with over-training and on downstream tasks](https://mlsanity.com/paper/2403.08540/) (2024)
- [OLMES: A Standard for Language Model Evaluations](https://mlsanity.com/paper/2406.08446/) (2024)
- [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://mlsanity.com/paper/2403.15796/) (2024)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Training Trajectories of Language Models Across Scales](https://mlsanity.com/paper/2212.09803/) (2022)
- [Inverse scaling can become U-shaped](https://mlsanity.com/paper/2211.02011/) (2022)
- [Paloma: A Benchmark for Evaluating Language Model Fit](https://mlsanity.com/paper/2312.10523/) (2023)
- [Scaling Laws for Predicting Downstream Performance in LLMs](https://mlsanity.com/paper/2410.08527/) (2024)
- [Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?](https://mlsanity.com/paper/2406.04391/) (2024)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale](https://mlsanity.com/paper/2409.17115/) (2024)
- [Establishing Task Scaling Laws via Compute-Efficient Model Ladders](https://mlsanity.com/paper/2412.04403/) (2024)
- [Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments](https://mlsanity.com/paper/2202.06387/) (2022)
- [How Much More Data Do I Need? Estimating Requirements for Downstream Tasks](https://mlsanity.com/paper/2207.01725/) (2022)

## Built on by (most cited first, 29 of 29)

- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [Train-before-Test Harmonizes Language Model Rankings](https://mlsanity.com/paper/2507.05195/) (2025)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [What do Language Models Learn and When? The Implicit Curriculum Hypothesis](https://mlsanity.com/paper/2604.08510/) (2026)
- [MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging](https://mlsanity.com/paper/2601.17858/) (2026)
- [Relative Scaling Laws for LLMs](https://mlsanity.com/paper/2510.24626/) (2025)
- [Data Scaling Laws for Radiology Foundation Models](https://mlsanity.com/paper/2509.12818/) (2025)
- [TESSERA v2: Scaling Pixel-wise Earth Foundation Models](https://mlsanity.com/paper/2607.03949/) (2026)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence](https://mlsanity.com/paper/2606.12058/) (2026)
- [Neural Neural Scaling Laws](https://mlsanity.com/paper/2601.19831/) (2026)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation](https://mlsanity.com/paper/2606.07616/) (2026)
- [Small-Scale Experiments: Are We There Yet?](https://mlsanity.com/paper/2608.11859/) (2026)
- [TailSFT: Filtered Fine-Tuning Improves Post-Training Performance](https://mlsanity.com/paper/2608.25756/) (2026)
- [Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks](https://mlsanity.com/paper/2508.18672/) (2025)
- [What Scales in Cross-Entropy Scaling Law?](https://mlsanity.com/paper/2510.04067/) (2025)
- [The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuning](https://mlsanity.com/paper/2602.11217/) (2026)
- [How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?](https://mlsanity.com/paper/2602.11618/) (2026)
- [Prescriptive Scaling Reveals the Evolution of Language Model Capabilities](https://mlsanity.com/paper/2602.15327/) (2026)
- [Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization](https://mlsanity.com/paper/2603.08022/) (2026)
- [Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor](https://mlsanity.com/paper/2605.20798/) (2026)
- [A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents](https://mlsanity.com/paper/2607.17205/) (2026)
- [Post-Training Science for Supervised Fine-Tuning](https://mlsanity.com/paper/2609.01244/) (2026)
- [Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI](https://mlsanity.com/paper/2603.25033/) (2026)
- [Provable Data Scaling Law for Meta Learning via Complexity Minimization](https://mlsanity.com/paper/2606.02008/) (2026)
- [Validity Threats for Foundation Model Research](https://mlsanity.com/paper/2606.05029/) (2026)
- [When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning](https://mlsanity.com/paper/2608.31118/) (2026)
- [Capability Scaling-Down Laws for LLM Compression](https://mlsanity.com/paper/2610.02462/) (2026)

## Abstract (arXiv)

Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple transformations of the performance metric, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, and we find that predictable scaling only occurs in a minority of cases: 39% of the time. Moreover, seemingly benign changes to the experimental setting can completely change the scaling behavior. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To accurately model the relationship between pretraining loss and task performance, we must embrace the cases in which scaling behavior deviates from linear trends.