# From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction

arXiv: https://arxiv.org/abs/2310.16802 · HTML: https://mlsanity.com/paper/2310.16802/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 70 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts](https://mlsanity.com/paper/2206.08917/) (2022)
- [SPICE, A Dataset of Drug-like Molecules and Peptides for Training Machine Learning Potentials](https://mlsanity.com/paper/2209.10702/) (2022)
- [Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching](https://mlsanity.com/paper/2206.13602/) (2022)
- [In Defense of the Unitary Scalarization for Deep Multi-Task Learning](https://mlsanity.com/paper/2201.04122/) (2022)
- [GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets](https://mlsanity.com/paper/2204.02782/) (2022)
- [An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storage](https://mlsanity.com/paper/2010.09435/) (2020)
- [Unified 2D and 3D Pre-Training of Molecular Representations](https://mlsanity.com/paper/2207.08806/) (2022)
- [Transition1x -- a Dataset for Building Generalizable Reactive Machine Learning Potentials](https://mlsanity.com/paper/2207.12858/) (2022)
- [Fractional Denoising for 3D Molecular Pre-training](https://mlsanity.com/paper/2307.10683/) (2023)
- [Artificial Intelligence in Drug Discovery: Applications and Techniques](https://mlsanity.com/paper/2106.05386/) (2021)
- [MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction](https://mlsanity.com/paper/2210.14188/) (2022)

## Built on by (most cited first, 30 of 39)

- [UMA: A Family of Universal Models for Atoms](https://mlsanity.com/paper/2506.23971/) (2025)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [All-atom Diffusion Transformers: Unified generative modelling of molecules and materials](https://mlsanity.com/paper/2503.03965/) (2025)
- [Orb: A Fast, Scalable Neural Network Potential](https://mlsanity.com/paper/2410.22570/) (2024)
- [Fully-inductive Node Classification on Arbitrary Graphs](https://mlsanity.com/paper/2405.20445/) (2024)
- [Doob's Lagrangian: A Sample-Efficient Variational Approach to Transition Path Sampling](https://mlsanity.com/paper/2410.07974/) (2024)
- [A practical guide to machine learning interatomic potentials -- Status and future](https://mlsanity.com/paper/2503.09814/) (2025)
- [textttMiniMol: A Parameter-Efficient Foundation Model for Molecular Learning](https://mlsanity.com/paper/2404.14986/) (2024)
- [AdsorbDiff: Adsorbate Placement via Conditional Denoising Diffusion](https://mlsanity.com/paper/2405.03962/) (2024)
- [MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Models](https://mlsanity.com/paper/2502.20933/) (2025)
- [MatterTune: An Integrated, User-Friendly Platform for Fine-Tuning Atomistic Foundation Models to Accelerate Materials Simulation and Discovery](https://mlsanity.com/paper/2504.10655/) (2025)
- [LeMat-Traj: A Scalable and Unified Dataset of Materials Trajectories for Atomistic Modeling](https://mlsanity.com/paper/2508.20875/) (2025)
- [Self-Conditioned Denoising for Atomistic Representation Learning](https://mlsanity.com/paper/2603.17196/) (2026)
- [Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling](https://mlsanity.com/paper/2410.08024/) (2024)
- [Understanding and Mitigating Distribution Shifts For Machine Learning Force Fields](https://mlsanity.com/paper/2503.08674/) (2025)
- [DEQuify your force field: More efficient simulations using deep equilibrium models](https://mlsanity.com/paper/2509.08734/) (2025)
- [Towards Agentic Intelligence for Materials Science](https://mlsanity.com/paper/2602.00169/) (2026)
- [Enhancing Machine Learning Potentials through Transfer Learning across Chemical Elements](https://mlsanity.com/paper/2502.13522/) (2025)
- [Better without U: Impact of Selective Hubbard U Correction on Foundational MLIPs](https://mlsanity.com/paper/2601.21056/) (2026)
- [Pushing the Pareto front of band gap and permittivity: ML-guided search for dielectric materials](https://mlsanity.com/paper/2401.05848/) (2024)
- [Pre-training Graph Neural Networks with Structural Fingerprints for Materials Discovery](https://mlsanity.com/paper/2503.01227/) (2025)
- [Accelerating and enhancing thermodynamic simulations of electrochemical interfaces](https://mlsanity.com/paper/2503.17870/) (2025)
- [Optimizing Data Distribution and Kernel Performance for Efficient Training of Chemistry Foundation Models: A Case Study with MACE](https://mlsanity.com/paper/2504.10700/) (2025)
- [Towards Faster and More Compact Foundation Models for Molecular Property Prediction](https://mlsanity.com/paper/2504.19538/) (2025)
- [Multi-task parallelism for robust pre-training of graph foundation models on multi-source, multi-fidelity atomistic modeling data](https://mlsanity.com/paper/2506.21788/) (2025)
- [Training Dynamics of Learning 3D-Rotational Equivariance](https://mlsanity.com/paper/2512.02303/) (2025)
- [Mixture of experts architectures for machine learning interatomic potentials](https://mlsanity.com/paper/2603.07977/) (2026)
- [BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models](https://mlsanity.com/paper/2605.08110/) (2026)
- [Improving Molecular Modeling with Geometric GNNs: an Empirical Study](https://mlsanity.com/paper/2407.08313/) (2024)
- [Implicit Delta Learning of High Fidelity Neural Network Potentials](https://mlsanity.com/paper/2412.06064/) (2024)

## Abstract (arXiv)

Foundation models have been transformational in machine learning fields such as natural language processing and computer vision. Similar success in atomic property prediction has been limited due to the challenges of training effective models across multiple chemical domains. To address this, we introduce Joint Multi-domain Pre-training (JMP), a supervised pre-training strategy that simultaneously trains on multiple datasets from different chemical domains, treating each dataset as a unique pre-training task within a multi-task framework. Our combined training dataset consists of ~120M systems from OC20, OC22, ANI-1x, and Transition-1x. We evaluate performance and generalization by fine-tuning over a diverse set of downstream tasks and datasets including: QM9, rMD17, MatBench, QMOF, SPICE, and MD22. JMP demonstrates an average improvement of 59% over training from scratch, and matches or sets state-of-the-art on 34 out of 40 tasks. Our work highlights the potential of pre-training strategies that utilize diverse data to advance property prediction across chemical domains, especially for low-data tasks. Please visit https://nima.sh/jmp for further information.