# EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations

arXiv: https://arxiv.org/abs/2310.02428 · HTML: https://mlsanity.com/paper/2310.02428/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [The Design Space of E(3)-Equivariant Atom-Centered Interatomic Potentials](https://mlsanity.com/paper/2205.06643/) (2022)
- [Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations](https://mlsanity.com/paper/2210.07237/) (2022)
- [The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Science](https://mlsanity.com/paper/2210.17484/) (2022)
- [MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling](https://mlsanity.com/paper/2309.05934/) (2023)

## Built on by (most cited first, 15 of 15)

- [A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems](https://mlsanity.com/paper/2312.07511/) (2023)
- [Foundational Large Language Models for Materials Research](https://mlsanity.com/paper/2412.09560/) (2024)
- [TorchMD-Net 2.0: Fast Neural Network Potentials for Molecular Simulations](https://mlsanity.com/paper/2402.17660/) (2024)
- [A Recipe for Charge Density Prediction](https://mlsanity.com/paper/2405.19276/) (2024)
- [Stability-Aware Training of Machine Learning Force Fields with Differentiable Boltzmann Estimators](https://mlsanity.com/paper/2402.13984/) (2024)
- [Transformers Discover Molecular Structure Without Graph Priors](https://mlsanity.com/paper/2510.02259/) (2025)
- [UniFFBench: Evaluating Universal Machine Learning Force Fields Against Experimental Measurements](https://mlsanity.com/paper/2508.05762/) (2025)
- [CrystalGym: A New Benchmark for Materials Discovery Using Reinforcement Learning](https://mlsanity.com/paper/2509.23156/) (2025)
- [Understanding and Mitigating Distribution Shifts For Machine Learning Force Fields](https://mlsanity.com/paper/2503.08674/) (2025)
- [Predicting and Interpreting Energy Barriers of Metallic Glasses with Graph Neural Networks](https://mlsanity.com/paper/2401.08627/) (2023)
- [Deconstructing equivariant representations in molecular systems](https://mlsanity.com/paper/2410.08131/) (2024)
- [ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamics](https://mlsanity.com/paper/2510.05482/) (2025)
- [Improving the Stability of GNN Force Field Models by Reducing Feature Correlation](https://mlsanity.com/paper/2502.12548/) (2025)
- [CrystalFramer: Rethinking the Role of Frames for SE(3)-Invariant Crystal Structure Modeling](https://mlsanity.com/paper/2503.02209/) (2025)
- [Graph Mamba Operator: A Latent Simulator for Interacting Particle Systems](https://mlsanity.com/paper/2606.09432/) (2026)

## Abstract (arXiv)

Equivariant graph neural networks force fields (EGraFFs) have shown great promise in modelling complex interactions in atomic systems by exploiting the graphs' inherent symmetries. Recent works have led to a surge in the development of novel architectures that incorporate equivariance-based inductive biases alongside architectural innovations like graph transformers and message passing to model atomic interactions. However, thorough evaluations of these deploying EGraFFs for the downstream task of real-world atomistic simulations, is lacking. To this end, here we perform a systematic benchmarking of 6 EGraFF algorithms (NequIP, Allegro, BOTNet, MACE, Equiformer, TorchMDNet), with the aim of understanding their capabilities and limitations for realistic atomistic simulations. In addition to our thorough evaluation and analysis on eight existing datasets based on the benchmarking literature, we release two new benchmark datasets, propose four new metrics, and three challenging tasks. The new datasets and tasks evaluate the performance of EGraFF to out-of-distribution data, in terms of different crystal structures, temperatures, and new molecules. Interestingly, evaluation of the EGraFF models based on dynamic simulations reveals that having a lower error on energy or force does not guarantee stable or reliable simulation or faithful replication of the atomic structures. Moreover, we find that no model clearly outperforms other models on all datasets and tasks. Importantly, we show that the performance of all the models on out-of-distribution datasets is unreliable, pointing to the need for the development of a foundation model for force fields that can be used in real-world simulations. In summary, this work establishes a rigorous framework for evaluating machine learning force fields in the context of atomic simulations and points to open research challenges within this domain.