# The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains

arXiv: https://arxiv.org/abs/2410.24169 · HTML: https://mlsanity.com/paper/2410.24169/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 58 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Machine Learning Force Fields](https://mlsanity.com/paper/2010.07067/) (2020)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [e3nn: Euclidean Neural Networks](https://mlsanity.com/paper/2207.09453/) (2022)
- [Spherical Message Passing for 3D Graph Networks](https://mlsanity.com/paper/2102.05013/) (2021)
- [Frame Averaging for Invariant and Equivariant Network Design](https://mlsanity.com/paper/2110.03336/) (2021)
- [Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs](https://mlsanity.com/paper/2302.03655/) (2023)
- [Approximately Equivariant Networks for Imperfectly Symmetric Dynamics](https://mlsanity.com/paper/2201.11969/) (2022)
- [The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts](https://mlsanity.com/paper/2206.08917/) (2022)
- [Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations](https://mlsanity.com/paper/2210.07237/) (2022)
- [SPICE, A Dataset of Drug-like Molecules and Peptides for Training Machine Learning Potentials](https://mlsanity.com/paper/2209.10702/) (2022)
- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)
- [GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets](https://mlsanity.com/paper/2204.02782/) (2022)
- [Spherical Channels for Modeling Atomic Interactions](https://mlsanity.com/paper/2206.14331/) (2022)
- [Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Space](https://mlsanity.com/paper/2310.02970/) (2023)
- [So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems](https://mlsanity.com/paper/2205.14276/) (2022)
- [Smooth, exact rotational symmetrization for deep learning on point clouds](https://mlsanity.com/paper/2305.19302/) (2023)
- [Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products](https://mlsanity.com/paper/2401.10216/) (2024)

## Built on by (most cited first, 30 of 40)

- [Learning Smooth and Expressive Interatomic Potentials for Physical Property Prediction](https://mlsanity.com/paper/2502.12147/) (2025)
- [The dark side of the forces: assessing non-conservative force models for atomistic machine learning](https://mlsanity.com/paper/2412.11569/) (2024)
- [An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks](https://mlsanity.com/paper/2501.13986/) (2025)
- [Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians](https://mlsanity.com/paper/2501.09009/) (2025)
- [Probing Equivariance and Symmetry Breaking in Convolutional Networks](https://mlsanity.com/paper/2501.01999/) (2025)
- [E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Products](https://mlsanity.com/paper/2501.19216/) (2025)
- [U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster](https://mlsanity.com/paper/2604.09041/) (2026)
- [Transferable Generative Models Bridge Femtosecond to Nanosecond Time-Step Molecular Dynamics](https://mlsanity.com/paper/2510.07589/) (2025)
- [Pushing the limits of unconstrained machine-learned interatomic potentials](https://mlsanity.com/paper/2601.16195/) (2026)
- [How simple can you go? An off-the-shelf transformer approach to molecular dynamics](https://mlsanity.com/paper/2503.01431/) (2025)
- [Platonic Transformers: A Solid Choice For Equivariance](https://mlsanity.com/paper/2510.03511/) (2025)
- [A universal augmentation framework for long-range electrostatics in machine learning interatomic potentials](https://mlsanity.com/paper/2507.14302/) (2025)
- [A recipe for scalable attention-based MLIPs: unlocking long-range accuracy with all-to-all node attention](https://mlsanity.com/paper/2603.06567/) (2026)
- [Transformers Discover Molecular Structure Without Graph Priors](https://mlsanity.com/paper/2510.02259/) (2025)
- [MatRIS: Toward Reliable and Efficient Pretrained Machine Learning Interatomic Potentials](https://mlsanity.com/paper/2603.02002/) (2026)
- [Learning Inter-Atomic Potentials without Explicit Equivariance](https://mlsanity.com/paper/2510.00027/) (2025)
- [Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materials](https://mlsanity.com/paper/2602.22251/) (2026)
- [A Materials Foundation Model via Hybrid Invariant-Equivariant Architectures](https://mlsanity.com/paper/2503.05771/) (2025)
- [Scaling Laws and Symmetry, Evidence from Neural Force Fields](https://mlsanity.com/paper/2510.09768/) (2025)
- [AQCat25: Unlocking spin-aware, high-fidelity machine learning potentials for heterogeneous catalysis](https://mlsanity.com/paper/2510.22938/) (2025)
- [Understanding and Mitigating Distribution Shifts For Machine Learning Force Fields](https://mlsanity.com/paper/2503.08674/) (2025)
- [Equivariant Spherical Transformer for Efficient Molecular Modeling](https://mlsanity.com/paper/2505.23086/) (2025)
- [E2Former-V2: On-the-Fly Equivariant Attention with Linear Activation Memory](https://mlsanity.com/paper/2601.16622/) (2026)
- [Elign: Equivariant Diffusion Model Alignment from Foundational Machine Learning Force Fields](https://mlsanity.com/paper/2601.21985/) (2026)
- [Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials](https://mlsanity.com/paper/2607.02499/) (2026)
- [AlphaNet: Scaling Up Local-frame-based Atomistic Interatomic Potential](https://mlsanity.com/paper/2501.07155/) (2025)
- [Global Plane Waves From Local Gaussians: Periodic Charge Densities in a Blink](https://mlsanity.com/paper/2601.19966/) (2026)
- [Diagonal Symmetrization of Neural Network Solvers for the Many-Electron Schr"odinger Equation](https://mlsanity.com/paper/2502.05318/) (2025)
- [Tensor Decomposition Networks for Fast Machine Learning Interatomic Potential Computations](https://mlsanity.com/paper/2507.01131/) (2025)
- [Scaling Laws for Neural Material Models](https://mlsanity.com/paper/2509.21811/) (2025)

## Abstract (arXiv)

Scaling has been critical in improving model performance and generalization in machine learning. It involves how a model's performance changes with increases in model size or input data, as well as how efficiently computational resources are utilized to support this growth. Despite successes in other areas, the study of scaling in Neural Network Interatomic Potentials (NNIPs) remains limited. NNIPs act as surrogate models for ab initio quantum mechanical calculations. The dominant paradigm here is to incorporate many physical domain constraints into the model, such as rotational equivariance. We contend that these complex constraints inhibit the scaling ability of NNIPs, and are likely to lead to performance plateaus in the long run. In this work, we take an alternative approach and start by systematically studying NNIP scaling strategies. Our findings indicate that scaling the model through attention mechanisms is efficient and improves model expressivity. These insights motivate us to develop an NNIP architecture designed for scalability: the Efficiently Scaled Attention Interatomic Potential (EScAIP). EScAIP leverages a multi-head self-attention formulation within graph neural networks, applying attention at the neighbor-level representations. Implemented with highly-optimized attention GPU kernels, EScAIP achieves substantial gains in efficiency--at least 10x faster inference, 5x less memory usage--compared to existing NNIPs. EScAIP also achieves state-of-the-art performance on a wide range of datasets including catalysts (OC20 and OC22), molecules (SPICE), and materials (MPTrj). We emphasize that our approach should be thought of as a philosophy rather than a specific model, representing a proof-of-concept for developing general-purpose NNIPs that achieve better expressivity through scaling, and continue to scale efficiently with increased computational resources and training data.