Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
Not on the map. Why?
Abstract (arXiv)
Data curriculums have become central to successful LLM training, yet principles governing optimal data placement remain unclear. We introduce the *training re-evaluation curve (TREC)*, a diagnostic that retrospectively evaluates training batches *using the final model weights*. The TREC characterizes how well a trained model retains training data as a function of *when* the data was encountered during training. Analyzing TRECs for models from 111M to 3.9B parameters, we show that placing high-quality data at low points on the TREC significantly improves performance. Importantly, while a TREC is initially observable only after training, we demonstrate it can be *predicted in advance* from AdamW's implicit EMA coefficients, enabling proactive curriculum design. By predicting TRECs for published training recipes, we explain prior ablations and reveal suboptimal data placements. We also align high-quality data with TREC minima in order to improve continual pre-training of a 3.9B-parameter LLM trained on 900B tokens.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 86 of its 107 references that are in the map. First posted on arXiv 29 Sep 2025.
Builds on
86 of 107 references.
References · 86
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2020GLU Variants Improve Transformercited by 1,804
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
38 more
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
- 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
- 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
- 2022Quantifying Memorization Across Neural Language Modelscited by 763
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2013Generating Sequences With Recurrent Neural Networkscited by 554
- 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
- 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
- 20242 OLMo 2 Furiouscited by 340
- 2023Studying Large Language Model Generalization with Influence Functionscited by 326
- 2018An Empirical Model of Large-Batch Trainingcited by 281
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
- 2021Hash Layers For Large Sparse Modelscited by 237
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
- 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
- 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
- 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
- 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
- 2023Understanding plasticity in neural networkscited by 174
- 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
- 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
- 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
- 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
- 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
- 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
- 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
- 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
- 2024Scaling Laws for Precisioncited by 95
- 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
- 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
- 2024How to Train Data-Efficient LLMscited by 93
- 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
50 most cited of 86 All 86 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.