# Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training

arXiv: https://arxiv.org/abs/2512.08894 · HTML: https://mlsanity.com/paper/2512.08894/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 9 Dec 2025.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [Language models scale reliably with over-training and on downstream tasks](https://mlsanity.com/paper/2403.08540/) (2024)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Understanding Emergent Abilities of Language Models from the Loss Perspective](https://mlsanity.com/paper/2403.15796/) (2024)
- [How Does Critical Batch Size Scale in Pre-training?](https://mlsanity.com/paper/2410.21676/) (2024)
- [Scaling Laws for Predicting Downstream Performance in LLMs](https://mlsanity.com/paper/2410.08527/) (2024)
- [Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training](https://mlsanity.com/paper/2505.13738/) (2025)
- [Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?](https://mlsanity.com/paper/2406.04391/) (2024)
- [DataDecide: How to Predict Best Pretraining Data with Small Experiments](https://mlsanity.com/paper/2504.11393/) (2025)
- [Establishing Task Scaling Laws via Compute-Efficient Model Ladders](https://mlsanity.com/paper/2412.04403/) (2024)
- [Loss-to-Loss Prediction: Scaling Laws for All Datasets](https://mlsanity.com/paper/2411.12925/) (2024)
- [Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit](https://mlsanity.com/paper/2410.05838/) (2024)

## Built on by (most cited first, 6 of 6)

- [Neural Neural Scaling Laws](https://mlsanity.com/paper/2601.19831/) (2026)
- [Prescriptive Scaling Reveals the Evolution of Language Model Capabilities](https://mlsanity.com/paper/2602.15327/) (2026)
- [To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval](https://mlsanity.com/paper/2604.00715/) (2026)
- [Scaling Properties of Continuous Diffusion Spoken Language Models](https://mlsanity.com/paper/2604.24416/) (2026)
- [LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss](https://mlsanity.com/paper/2602.12005/) (2026)
- [Validity Threats for Foundation Model Research](https://mlsanity.com/paper/2606.05029/) (2026)

## Abstract (arXiv)

While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of benchmark performance from the training budget. We find that for a fixed token-to-parameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure, which is prone to compounding errors. Furthermore, we introduce functional forms that predict accuracy across token-to-parameter ratios and account for inference compute under repeated sampling. We validate our findings on models with up to 17B parameters trained on up to 350B tokens across two dataset mixtures. To support reproducibility and encourage future research, we release the complete set of pretraining losses and downstream evaluation results.