# Synthetic pre-training for neural-network interatomic potentials

arXiv: https://arxiv.org/abs/2307.15714 · HTML: https://mlsanity.com/paper/2307.15714/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [The Design Space of E(3)-Equivariant Atom-Centered Interatomic Potentials](https://mlsanity.com/paper/2205.06643/) (2022)
- [Two for One: Diffusion Models and Force Fields for Coarse-Grained Molecular Dynamics](https://mlsanity.com/paper/2302.00600/) (2023)
- [TensorNet: Cartesian Tensor Representations for Efficient Learning of Molecular Potentials](https://mlsanity.com/paper/2306.06482/) (2023)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [DPA-1: Pretraining of Attention-based Deep Potential Model for Molecular Simulation](https://mlsanity.com/paper/2208.08236/) (2022)
- [Better Language Models of Code through Self-Improvement](https://mlsanity.com/paper/2304.01228/) (2023)
- [Transfer learning for chemically accurate interatomic neural network potentials](https://mlsanity.com/paper/2212.03916/) (2022)
- [Accelerating Molecular Graph Neural Networks via Knowledge Distillation](https://mlsanity.com/paper/2306.14818/) (2023)
- [Injecting Domain Knowledge from Empirical Interatomic Potentials to Neural Networks for Predicting Material Properties](https://mlsanity.com/paper/2210.08047/) (2022)

## Built on by (most cited first, 6 of 6)

- [HIP: Hessian Interatomic Potentials without derivatives](https://mlsanity.com/paper/2509.21624/) (2025)
- [Understanding and Mitigating Distribution Shifts For Machine Learning Force Fields](https://mlsanity.com/paper/2503.08674/) (2025)
- [Enhancing Machine Learning Potentials through Transfer Learning across Chemical Elements](https://mlsanity.com/paper/2502.13522/) (2025)
- [Implicit Delta Learning of High Fidelity Neural Network Potentials](https://mlsanity.com/paper/2412.06064/) (2024)
- [Ensemble Knowledge Distillation for Machine Learning Interatomic Potentials](https://mlsanity.com/paper/2503.14293/) (2025)
- [Distilling latent electrostatics from foundation machine learning interatomic potentials](https://mlsanity.com/paper/2606.15001/) (2026)

## Abstract (arXiv)

Machine learning (ML) based interatomic potentials have transformed the field of atomistic materials modelling. However, ML potentials depend critically on the quality and quantity of quantum-mechanical reference data with which they are trained, and therefore developing datasets and training pipelines is becoming an increasingly central challenge. Leveraging the idea of "synthetic" (artificial) data that is common in other areas of ML research, we here show that synthetic atomistic data, themselves obtained at scale with an existing ML potential, constitute a useful pre-training task for neural-network interatomic potential models. Once pre-trained with a large synthetic dataset, these models can be fine-tuned on a much smaller, quantum-mechanical one, improving numerical accuracy and stability in computational practice. We demonstrate feasibility for a series of equivariant graph-neural-network potentials for carbon, and we carry out initial experiments to test the limits of the approach.