mlsanityData to 14 Oct 2026

ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models

arXiv 2505.12534 ↗First posted 18 May 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Foundation models have shown remarkable success across scientific domains, yet their impact in chemistry remains limited due to the absence of diverse, large-scale, high-quality datasets that reflect the field's multifaceted nature. We present the ChemPile, an open dataset containing over 75 billion tokens of curated chemical data, specifically built for training and evaluating general-purpose models in the chemical sciences. The dataset mirrors the human learning journey through chemistry -- from educational foundations to specialized expertise -- spanning multiple modalities and content types including structured data in diverse chemical representations (SMILES, SELFIES, IUPAC names, InChI, molecular renderings), scientific and educational text, executable code, and chemical images. ChemPile integrates foundational knowledge (textbooks, lecture notes), specialized expertise (scientific articles and language-interfaced data), visual understanding (molecular structures, diagrams), and advanced reasoning (problem-solving traces and code) -- mirroring how human chemists develop expertise through diverse learning materials and experiences. Constructed through hundreds of hours of expert curation, the ChemPile captures both foundational concepts and domain-specific complexity. We provide standardized training, validation, and test splits, enabling robust benchmarking. ChemPile is openly released via HuggingFace with a consistent API, permissive license, and detailed documentation. We hope the ChemPile will serve as a catalyst for chemical AI, enabling the development of the next generation of chemical foundation models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 53 of its 88 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

53 of 88 references.

References · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  11. 2023StarCoder: may the source be with you!cited by 799
  12. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
38 more
  1. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  2. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  3. 2023Textbooks Are All You Needcited by 569
  4. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  5. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  6. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  7. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  8. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  9. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  10. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  11. 2024A Survey on Data Selection for Language Modelscited by 236
  12. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  13. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 166
  14. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  15. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  16. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  17. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  18. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  19. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  20. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  21. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 93
  22. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  23. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  24. 2022SELFIES and the future of molecular string representationscited by 76
  25. 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 61
  26. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  27. 2023Can Large Language Models Empower Molecular Property Prediction?cited by 44
  28. 2023MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalitiescited by 38
  29. 2024Are large language models superhuman chemists?cited by 34
  30. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 32
  31. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  32. 2024Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planningcited by 29
  33. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 26
  34. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  35. 2024Benchmarking Large Language Models for Molecule Prediction Taskscited by 17
  36. 2023An Interdisciplinary Outlook on Large Language Models for Scientific Researchcited by 15
  37. 2022Machine Learning for a Sustainable Energy Futurecited by 14
  38. 2024BatGPT-Chem: A Foundation Large Model For Retrosynthesis Predictioncited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Molecular Machine Learning in Chemical Process Designcited by 5
  2. 2025BioMedGPT-Mol: Multi-task Learning for Molecular Understanding and Generationcited by 3
  3. 2025Uni-Parser Technical Reportcited by 3
  4. 2026Segment-Level Coherence for Robust Harmful Intent Probing in LLMscited by 3
  5. 2026Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Languagecited by 1
  6. 2026Semantic Content Determines Algorithmic Performancecited by 0
  7. 2026Beyond Learning on Molecules by Weakly Supervising on Moleculescited by 0
  8. 2026Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Miningcited by 0
  9. 2026SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generationcited by 0