# DPA-1: Pretraining of Attention-based Deep Potential Model for Molecular Simulation

arXiv: https://arxiv.org/abs/2208.08236 · HTML: https://mlsanity.com/paper/2208.08236/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 77 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [Attention in Natural Language Processing](https://mlsanity.com/paper/1902.02181/) (2019)
- [Less is more: sampling chemical space with active learning](https://mlsanity.com/paper/1801.09319/) (2018)
- [GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets](https://mlsanity.com/paper/2204.02782/) (2022)
- [Rotation Invariant Graph Neural Networks using Spin Convolutions](https://mlsanity.com/paper/2106.09575/) (2021)
- [Spherical Channels for Modeling Atomic Interactions](https://mlsanity.com/paper/2206.14331/) (2022)
- [Active Learning of Uniformly Accurate Inter-atomic Potentials for Materials Simulation](https://mlsanity.com/paper/1810.11890/) (2018)
- [Equivariant Graph Attention Networks for Molecular Property Prediction](https://mlsanity.com/paper/2202.09891/) (2022)

## Built on by (most cited first, 15 of 15)

- [MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Models](https://mlsanity.com/paper/2502.20933/) (2025)
- [Synthetic pre-training for neural-network interatomic potentials](https://mlsanity.com/paper/2307.15714/) (2023)
- [Overcoming systematic softening in universal machine learning interatomic potentials by fine-tuning](https://mlsanity.com/paper/2405.07105/) (2024)
- [Transfer Learning for Molecular Property Predictions from Small Data Sets](https://mlsanity.com/paper/2404.13393/) (2024)
- [AlphaNet: Scaling Up Local-frame-based Atomistic Interatomic Potential](https://mlsanity.com/paper/2501.07155/) (2025)
- [Scalable Machine Learning Force Fields for Macromolecular Systems Through Long-Range Aware Message Passing](https://mlsanity.com/paper/2601.03774/) (2026)
- [Better without U: Impact of Selective Hubbard U Correction on Foundational MLIPs](https://mlsanity.com/paper/2601.21056/) (2026)
- [The OpenLAM Challenges](https://mlsanity.com/paper/2501.16358/) (2025)
- [ADAPT: Lightweight, Long-Range Machine Learning Force Fields Without Graphs](https://mlsanity.com/paper/2509.24115/) (2025)
- [Mixture of experts architectures for machine learning interatomic potentials](https://mlsanity.com/paper/2603.07977/) (2026)
- [Pretraining Strategy for Neural Potentials](https://mlsanity.com/paper/2402.15921/) (2024)
- [A General Neural Network Potential for Energetic Materials with C, H, N, and O elements](https://mlsanity.com/paper/2503.01932/) (2025)
- [Are Foundational Atomistic Models Reliable for Finite-Temperature Molecular Dynamics?](https://mlsanity.com/paper/2503.08207/) (2025)
- [Learning task-specific predictive models for scientific computing](https://mlsanity.com/paper/2506.03835/) (2025)
- [Making Room for AI: Multi-GPU Molecular Dynamics with Deep Potentials in GROMACS](https://mlsanity.com/paper/2604.07276/) (2026)

## Abstract (arXiv)

Machine learning assisted modeling of the inter-atomic potential energy surface (PES) is revolutionizing the field of molecular simulation. With the accumulation of high-quality electronic structure data, a model that can be pretrained on all available data and finetuned on downstream tasks with a small additional effort would bring the field to a new stage. Here we propose DPA-1, a Deep Potential model with a novel attention mechanism, which is highly effective for representing the conformation and chemical spaces of atomic systems and learning the PES. We tested DPA-1 on a number of systems and observed superior performance compared with existing benchmarks. When pretrained on large-scale datasets containing 56 elements, DPA-1 can be successfully applied to various downstream tasks with a great improvement of sample efficiency. Surprisingly, for different elements, the learned type embedding parameters form a spiral in the latent space and have a natural correspondence with their positions on the periodic table, showing interesting interpretability of the pretrained DPA-1 model.