mlsanityData to 7 Oct 2026

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

arXiv 2412.15285 ↗First posted 18 Dec 2024cs.CL cs.AI cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Pretraining large language models effectively requires strategic data selection, blending and ordering. However, key details about data mixtures especially their scalability to longer token horizons and larger model sizes remain underexplored due to limited disclosure by model developers. To address this, we formalize the concept of two-phase pretraining and conduct an extensive systematic study on how to select and mix data to maximize model accuracies for the two phases. Our findings illustrate that a two-phase approach for pretraining outperforms random data ordering and natural distribution of tokens by 3.4% and 17% on average accuracies. We provide in-depth guidance on crafting optimal blends based on quality of the data source and the number of epochs to be seen. We propose to design blends using downsampled data at a smaller scale of 1T tokens and then demonstrate effective scaling of our approach to larger token horizon of 15T tokens and larger model size of 25B model size. These insights provide a series of steps practitioners can follow to design and scale their data blends.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

44 of 61 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
32 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  12. 2023StarCoder: may the source be with you!cited by 790
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  14. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  15. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  16. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  17. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  18. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  19. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  20. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  21. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  22. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  23. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  24. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  25. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  26. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  27. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  28. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  29. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  30. 2024Is Child-Directed Speech Effective Training Data for Language Models?cited by 24
  31. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 22
  32. 2023CLIMB: Curriculum Learning for Infant-inspired Model Buildingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 7.

24 papers.

Sort:

By area:

  1. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  2. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  3. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  4. 2025Pretraining Large Language Models with NVFP4cited by 67
  5. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  6. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  7. 2025A Survey on LLM Mid-Trainingcited by 18
  8. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 15
  9. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  10. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 10
  11. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  12. 2025Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Qualitycited by 7
12 more
  1. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 7
  2. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  3. 2025A Survey of LLM × DATAcited by 4
  4. 2025Curriculum-Guided Layer Scaling for Language Model Pretrainingcited by 4
  5. 2025Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asiacited by 3
  6. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  7. 2026RegMix-D: Dynamic Data Mixing via Proxy Training Trajectoriescited by 2
  8. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrievalcited by 1
  9. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 0
  10. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoningcited by 0
  11. 2026Understanding Data Temporality Impact on Large Language Models Pre-trainingcited by 0
  12. 2026Sequential Pretraining Favors Large Modelscited by 0