# Ensemble Knowledge Distillation for Machine Learning Interatomic Potentials

arXiv: https://arxiv.org/abs/2503.14293 · HTML: https://mlsanity.com/paper/2503.14293/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 65 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [Less is more: sampling chemical space with active learning](https://mlsanity.com/paper/1801.09319/) (2018)
- [A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems](https://mlsanity.com/paper/2312.07511/) (2023)
- [Hierarchical modeling of molecular energies using a deep neural network](https://mlsanity.com/paper/1710.00017/) (2017)
- [Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians](https://mlsanity.com/paper/2501.09009/) (2025)
- [A predictive machine learning force field framework for liquid electrolyte development](https://mlsanity.com/paper/2404.07181/) (2024)
- [Automated discovery of a robust interatomic potential for aluminum](https://mlsanity.com/paper/2003.04934/) (2020)
- [Data efficiency and extrapolation trends in neural network interatomic potentials](https://mlsanity.com/paper/2302.05823/) (2023)
- [Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN](https://mlsanity.com/paper/2406.12909/) (2024)
- [Synthetic pre-training for neural-network interatomic potentials](https://mlsanity.com/paper/2307.15714/) (2023)
- [Synthetic data enable experiments in atomistic machine learning](https://mlsanity.com/paper/2211.16443/) (2022)
- [Teacher-student training improves accuracy and efficiency of machine learning interatomic potentials](https://mlsanity.com/paper/2502.05379/) (2025)

## Abstract (arXiv)

The quality of machine learning interatomic potentials (MLIPs) strongly depends on the quantity of training data as well as the quantum chemistry (QC) level of theory used. Datasets generated with high-fidelity QC methods are typically restricted to small molecules and may be missing energy gradients, which make it difficult to train accurate MLIPs. We present an ensemble knowledge distillation (EKD) method to improve MLIP accuracy when trained to energy-only datasets. First, multiple teacher models are trained to QC energies and then generate atomic forces for all configurations in the dataset. Next, the student MLIP is trained to both QC energies and to ensemble-averaged forces generated by the teacher models. We apply this workflow on the ANI-1ccx dataset where the configuration energies computed at the coupled cluster level of theory. The resulting student MLIPs achieve new state-of-the-art accuracy on the COMP6 benchmark and show improved stability for molecular dynamics simulations.