mlsanityData to 14 Oct 2026

Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training

arXiv 2602.07824 ↗First posted 8 Feb 2026cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualizes data-model co-evolution: advanced models produce superior data for next-generation systems. We validate this on scientific literature by constructing Darwin-Science, a 900B-token corpus (L0-L5). We identify a learnability gap in raw scientific text, which we bridge via L4 (Generative Refinement) and L5 (Cognitive Completion) using frontier LLMs to explicate reasoning and terminology. To ensure rigorous attribution, we pre-trained daVinci-origin-3B/7B models from scratch, excluding scientific content to create contamination-free baselines. After 600B tokens of continued pre-training, Darwin-Science outperforms baselines by +2.12 (3B) and +2.95 (7B) points across 20+ benchmarks, rising to +5.60 and +8.40 points on domain-aligned tasks. Systematic progression to L5 yields a +1.36 total gain, confirming that higher-level processing unlocks latent data value. We release the Darwin-Science corpus and daVinci-origin models to enable principled, co-evolutionary development.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

43 of 65 references.

References · 43

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
31 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  10. 2022Galactica: A Large Language Model for Sciencecited by 744
  11. 2023Textbooks Are All You Needcited by 569
  12. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  13. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  14. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  15. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  16. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  17. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  18. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  19. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  20. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
  21. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 71
  22. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  23. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59
  24. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  25. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  26. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  27. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  28. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  29. 2024AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmarkcited by 24
  30. 2025RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programscited by 17
  31. 2025Generative Data Refinement: Just Ask for Better Datacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026DataFoundry: Evolving Data Preparators via Recursive Self-Improvement
  2. 2026ADE: Agentic Data Evolution Framework for Human-Centered Objectives
  3. 2025Evolution without an Oracle: Driving Effective Evolution with LLM Judges
  4. 2026InferenceEvolve: Towards Automated Causal Effect Estimators through Self-Evolving AI
  5. 2026EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation
  6. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science
  7. 2026Agentic-imodels: Evolving agentic interpretability tools via autoresearch
  8. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning