# LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

arXiv: https://arxiv.org/abs/2502.12120 · HTML: https://mlsanity.com/paper/2502.12120/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2025.

## Built on by (most cited first, 16 of 16)

- [Dynamic Chunking for End-to-End Hierarchical Sequence Modeling](https://mlsanity.com/paper/2507.07955/) (2025)
- [Scaling Laws for Optimal Data Mixtures](https://mlsanity.com/paper/2507.09404/) (2025)
- [Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training](https://mlsanity.com/paper/2509.26625/) (2025)
- [Language Models Improve When Pretraining Data Matches Target Tasks](https://mlsanity.com/paper/2507.12466/) (2025)
- [Train-before-Test Harmonizes Language Model Rankings](https://mlsanity.com/paper/2507.05195/) (2025)
- [MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model](https://mlsanity.com/paper/2510.11653/) (2025)
- [MentisOculi: Revealing the Limits of Reasoning with Mental Imagery](https://mlsanity.com/paper/2602.02465/) (2026)
- [Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation](https://mlsanity.com/paper/2505.00022/) (2025)
- [Weight Decay Improves Language Model Plasticity](https://mlsanity.com/paper/2602.11137/) (2026)
- [NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training](https://mlsanity.com/paper/2603.03597/) (2026)
- [Membership Inference Attacks on Tokenizers of Large Language Models](https://mlsanity.com/paper/2510.05699/) (2025)
- [Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks](https://mlsanity.com/paper/2607.17624/) (2026)
- [Small-Scale Experiments: Are We There Yet?](https://mlsanity.com/paper/2608.11859/) (2026)
- [ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining](https://mlsanity.com/paper/2505.19893/) (2025)
- [LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure](https://mlsanity.com/paper/2608.13545/) (2026)
- [Relative Generalization Invariance of LLM Pretraining](https://mlsanity.com/paper/2609.33016/) (2026)

## Abstract (arXiv)

Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute. More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for understanding and improving LLM performance and generalization. In this work, we investigate which factors most strongly influence loss-to-loss scaling. Our experiments reveal that the pretraining data determines the scaling trend. In contrast, model size, optimization hyperparameters, tokenizer and even significant architectural differences, such as between transformer-based models like Llama and state-space models like Mamba, generally have limited impact. Consequently, practitioners should carefully curate suitable pretraining datasets for optimal downstream performance, while architectures and other settings can be freely optimized for training efficiency.