Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
Not on the map. Why?
Abstract (arXiv)
Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualizes data-model co-evolution: advanced models produce superior data for next-generation systems. We validate this on scientific literature by constructing Darwin-Science, a 900B-token corpus (L0-L5). We identify a learnability gap in raw scientific text, which we bridge via L4 (Generative Refinement) and L5 (Cognitive Completion) using frontier LLMs to explicate reasoning and terminology. To ensure rigorous attribution, we pre-trained daVinci-origin-3B/7B models from scratch, excluding scientific content to create contamination-free baselines. After 600B tokens of continued pre-training, Darwin-Science outperforms baselines by +2.12 (3B) and +2.95 (7B) points across 20+ benchmarks, rising to +5.60 and +8.40 points on domain-aligned tasks. Systematic progression to L5 yields a +1.36 total gain, confirming that higher-level processing unlocks latent data value. We release the Darwin-Science corpus and daVinci-origin models to enable principled, co-evolutionary development.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 8 Feb 2026.
Builds on
43 of 65 references.
References · 43
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Qwen Technical Reportcited by 3,344
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
31 more
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2022Galactica: A Large Language Model for Sciencecited by 744
- 2023Textbooks Are All You Needcited by 569
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
- 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
- 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
- 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
- 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
- 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
- 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 71
- 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
- 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59
- 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
- 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
- 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
- 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
- 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
- 2024AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmarkcited by 24
- 2025RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programscited by 17
- 2025Generative Data Refinement: Just Ask for Better Datacited by 6
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curationcited by 3
- 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
- 2026LaSR: Context-Aware Speech Recognition via Latent Reasoningcited by 1
- 2026EvoPool: Evolutionary Programmatic Annotation for Label-Efficient Specialized Supervisioncited by 1
- 2026DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Datacited by 1
- 2026daVinci-LLM:Towards the Science of Pretrainingcited by 0
- 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistantscited by 0
- 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants9 Sep 2026
- 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling5 Aug 2026
- 2026DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data27 Jul 2026
- 2026EvoPool: Evolutionary Programmatic Annotation for Label-Efficient Specialized Supervision1 Jun 2026
- 2026LaSR: Context-Aware Speech Recognition via Latent Reasoning30 May 2026
- 2026daVinci-LLM:Towards the Science of Pretraining28 Mar 2026
- 2026Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation15 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026DataFoundry: Evolving Data Preparators via Recursive Self-Improvement
- 2026ADE: Agentic Data Evolution Framework for Human-Centered Objectives
- 2025Evolution without an Oracle: Driving Effective Evolution with LLM Judges
- 2026InferenceEvolve: Towards Automated Causal Effect Estimators through Self-Evolving AI
- 2026EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation
- 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science
- 2026Agentic-imodels: Evolving agentic interpretability tools via autoresearch
- 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning