# PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets

arXiv: https://arxiv.org/abs/2510.23198 · HTML: https://mlsanity.com/paper/2510.23198/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 17 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster](https://mlsanity.com/paper/2304.03208/) (2023)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [Overtrained Language Models Are Harder to Fine-Tune](https://mlsanity.com/paper/2503.19206/) (2025)
- [As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages](https://mlsanity.com/paper/2012.05628/) (2020)
- [D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models](https://mlsanity.com/paper/2406.01375/) (2024)
- [Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection](https://mlsanity.com/paper/2502.06042/) (2025)
- [MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning](https://mlsanity.com/paper/2407.20999/) (2024)

## Abstract (arXiv)

Continual pre-training (CPT) for domain adaptation must balance target-domain gains with stability on the base domain. Existing CPT scaling laws typically assume a fixed pre-training budget, which limits their ability to forecast adaptation outcomes for models trained at different tokens-per-parameter (PTPP). We present PTPP-aware adaptation scaling laws that make the pre-training budget an explicit variable, enabling accurate prediction of adaptation loss at unseen ptpp. On a multilingual setup (English/Arabic rightarrow French), PTPP-aware formulations trained on early stages (ptpp=15,31) predict target loss at ptpp=279 and outperform a PTPP-agnostic dcpt transfer baseline on metrics (Huber-on-log, MAErel, calibration slope); full diagnostics (RMSE, MAPE) are in the appendix. Beyond forecasting, we show a practical use case: planning replay ratios and adaptation token budgets that satisfy target and forgetting constraints under compute limits.