# Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

arXiv: https://arxiv.org/abs/2610.05126 · HTML: https://mlsanity.com/paper/2610.05126/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 4 Oct 2026.

## Builds on (most cited first, 22 of 22)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance](https://mlsanity.com/paper/2403.16952/) (2024)
- [RegMix: Data Mixture as Regression for Language Model Pre-training](https://mlsanity.com/paper/2407.01492/) (2024)
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://mlsanity.com/paper/2205.10487/) (2022)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis](https://mlsanity.com/paper/2305.13230/) (2023)
- [Scaling Laws for Optimal Data Mixtures](https://mlsanity.com/paper/2507.09404/) (2025)
- [D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models](https://mlsanity.com/paper/2406.01375/) (2024)
- [DataDecide: How to Predict Best Pretraining Data with Small Experiments](https://mlsanity.com/paper/2504.11393/) (2025)
- [BiMix: A Bivariate Data Mixing Law for Language Model Pretraining](https://mlsanity.com/paper/2405.14908/) (2024)
- [AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs](https://mlsanity.com/paper/2407.20177/) (2024)
- [Prescriptive Scaling Laws for Data Constrained Training](https://mlsanity.com/paper/2605.01640/) (2026)
- [Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework](https://mlsanity.com/paper/2503.21023/) (2025)
- [Scaling Laws for Mixture Pretraining Under Data Constraints](https://mlsanity.com/paper/2605.12715/) (2026)
- [Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws](https://mlsanity.com/paper/2606.06888/) (2026)
- [Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them](https://mlsanity.com/paper/2606.07597/) (2026)
- [Scaling Domain Data Repetition in LLM Pretraining](https://mlsanity.com/paper/2608.14071/) (2026)
- [M3 Scaling Law: Optimizing Multi-Epoch, Multi-Lingual, and Multi-Stage Training for Low-Resource Language Models](https://mlsanity.com/paper/2410.12325/) (2024)

## Abstract (arXiv)

The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with model size, and a 520M Proof-Pile-2 experiment confirms that reducing repetition from sixteen to eight improves loss while using fewer training tokens. We use loss curves from several smaller models to retain a short list of promising repetition counts for evaluation at a larger scale. On PubMed and Caselaw, candidate sets fixed before target-model training retain the lowest-loss measured count on the original evaluation grids at both 200M and 520M. This supports candidate retention as a practical alternative to exact point prediction. We also relate the pruning regression to an empirical scaling model with two opposing repetition-dependent loss terms. A first-order expansion in log model size yields the linear form used by the selection rule, providing a scaling-based interpretation of the candidate-selection procedure.