mlsanityData to 14 Oct 2026

On the Importance of Pretraining Data Alignment for Atomic Property Prediction

arXiv 2502.11085 ↗First posted 16 Feb 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper challenges the recent paradigm in atomic property prediction that links progress to growing dataset sizes and computational resources. We show that pretraining on a carefully selected task-aligned dataset can match or even surpass large-scale joint pretraining while using only 1/24th of the pretraining budget. We introduce the Chemical Similarity Index (CSI), a simple metric for molecular graphs inspired by the Fr'echet Inception Distance in computer vision, which quantifies the alignment between upstream pretraining datasets and downstream tasks. By selecting the most aligned dataset with minimal CSI distance, we show that models pretrained on a smaller, focused dataset consistently achieve better performance on downstream tasks than those pretrained on massive, mixed datasets such as JMP. This holds even when the mixed dataset includes the upstream dataset most aligned with the downstream task. Counterintuitively, we also find that indiscriminately adding more data can degrade model performance when the additional data is poorly aligned with the target task. Our findings highlight that quality often outperforms quantity in pretraining for atomic property prediction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

37 of 58 references.

References · 37

  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  3. 2020Dataset Condensation with Gradient Matchingcited by 516
  4. 2021Dataset Condensation with Distribution Matchingcited by 365
  5. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  6. 2018GuacaMol: Benchmarking Models for De Novo Molecular Designcited by 297
  7. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  8. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  9. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  10. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  11. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  12. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
25 more
  1. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  2. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 204
  3. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 194
  4. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  5. 2021Graph Condensation for Graph Neural Networkscited by 150
  6. 2023Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNscited by 117
  7. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  8. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  9. 2022SPICE, A Dataset of Drug-like Molecules and Peptides for Training Machine Learning Potentialscited by 82
  10. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  11. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  12. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  13. 2022Graph Condensation via Receptive Field Distribution Matchingcited by 60
  14. 2023Real-Time Evaluation in Online Continual Learning: A New Hopecited by 47
  15. 2023From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Predictioncited by 39
  16. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  17. 2019Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraintscited by 37
  18. 2022Energy-Motivated Equivariant Pretraining for 3D Molecular Graphscited by 37
  19. 2022Transition1x -- a Dataset for Building Generalizable Reactive Machine Learning Potentialscited by 37
  20. 2023Internet Explorer: Targeted Representation Learning on the Open Webcited by 21
  21. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  22. 2019Budget-Aware Adapters for Multi-Domain Learningcited by 18
  23. 2024Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fieldscited by 18
  24. 2013Are all training examples equally valuable?cited by 14
  25. 2024On Pretraining Data Diversity for Self-Supervised Learningcited by 7

Built on by

Not cited yet.