mlsanityData to 14 Oct 2026

The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains

arXiv 2410.24169 ↗First posted 31 Oct 2024cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Scaling has been critical in improving model performance and generalization in machine learning. It involves how a model's performance changes with increases in model size or input data, as well as how efficiently computational resources are utilized to support this growth. Despite successes in other areas, the study of scaling in Neural Network Interatomic Potentials (NNIPs) remains limited. NNIPs act as surrogate models for ab initio quantum mechanical calculations. The dominant paradigm here is to incorporate many physical domain constraints into the model, such as rotational equivariance. We contend that these complex constraints inhibit the scaling ability of NNIPs, and are likely to lead to performance plateaus in the long run. In this work, we take an alternative approach and start by systematically studying NNIP scaling strategies. Our findings indicate that scaling the model through attention mechanisms is efficient and improves model expressivity. These insights motivate us to develop an NNIP architecture designed for scalability: the Efficiently Scaled Attention Interatomic Potential (EScAIP). EScAIP leverages a multi-head self-attention formulation within graph neural networks, applying attention at the neighbor-level representations. Implemented with highly-optimized attention GPU kernels, EScAIP achieves substantial gains in efficiency--at least 10x faster inference, 5x less memory usage--compared to existing NNIPs. EScAIP also achieves state-of-the-art performance on a wide range of datasets including catalysts (OC20 and OC22), molecules (SPICE), and materials (MPTrj). We emphasize that our approach should be thought of as a philosophy rather than a specific model, representing a proof-of-concept for developing general-purpose NNIPs that achieve better expressivity through scaling, and continue to scale efficiently with increased computational resources and training data.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 33 of its 58 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

33 of 58 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2021Scaling Vision Transformerscited by 999
  4. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  5. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
  6. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 571
  7. 2020Directional Message Passing for Molecular Graphscited by 561
  8. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 474
  9. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  10. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  11. 2020Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Moleculescited by 255
  12. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
21 more
  1. 2020Machine Learning Force Fieldscited by 207
  2. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 204
  3. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 182
  4. 2022e3nn: Euclidean Neural Networkscited by 174
  5. 2021Spherical Message Passing for 3D Graph Networkscited by 168
  6. 2021Frame Averaging for Invariant and Equivariant Network Designcited by 168
  7. 2023Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNscited by 117
  8. 2022Approximately Equivariant Networks for Imperfectly Symmetric Dynamicscited by 97
  9. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  10. 2022Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulationscited by 95
  11. 2022SPICE, A Dataset of Drug-like Molecules and Peptides for Training Machine Learning Potentialscited by 82
  12. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 75
  13. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  14. 2022Spherical Channels for Modeling Atomic Interactionscited by 47
  15. 2023Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Spacecited by 41
  16. 2022So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systemscited by 39
  17. 2023Smooth, exact rotational symmetrization for deep learning on point cloudscited by 39
  18. 2024Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Productscited by 36
  19. 2024Cartesian atomic cluster expansion for machine learning interatomic potentialscited by 31
  20. 2023Evaluation of the MACE Force Field Architecture: from Medicinal Chemistry to Materials Sciencecited by 25
  21. 2024TorchMD-Net 2.0: Fast Neural Network Potentials for Molecular Simulationscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 16.

40 papers.

Sort:

By area:

  1. 2025Learning Smooth and Expressive Interatomic Potentials for Physical Property Predictioncited by 84
  2. 2024The dark side of the forces: assessing non-conservative force models for atomistic machine learningcited by 24
  3. 2025An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networkscited by 22
  4. 2025Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessianscited by 15
  5. 2025Probing Equivariance and Symmetry Breaking in Convolutional Networkscited by 12
  6. 2025E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Productscited by 12
  7. 2026U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecastercited by 12
  8. 2025Transferable Generative Models Bridge Femtosecond to Nanosecond Time-Step Molecular Dynamicscited by 11
  9. 2026Pushing the limits of unconstrained machine-learned interatomic potentialscited by 11
  10. 2025How simple can you go? An off-the-shelf transformer approach to molecular dynamicscited by 10
  11. 2025Platonic Transformers: A Solid Choice For Equivariancecited by 10
  12. 2025A universal augmentation framework for long-range electrostatics in machine learning interatomic potentialscited by 9
28 more
  1. 2026A recipe for scalable attention-based MLIPs: unlocking long-range accuracy with all-to-all node attentioncited by 8
  2. 2025Transformers Discover Molecular Structure Without Graph Priorscited by 7
  3. 2026MatRIS: Toward Reliable and Efficient Pretrained Machine Learning Interatomic Potentialscited by 7
  4. 2025Learning Inter-Atomic Potentials without Explicit Equivariancecited by 5
  5. 2026Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materialscited by 5
  6. 2025A Materials Foundation Model via Hybrid Invariant-Equivariant Architecturescited by 4
  7. 2025Scaling Laws and Symmetry, Evidence from Neural Force Fieldscited by 4
  8. 2025AQCat25: Unlocking spin-aware, high-fidelity machine learning potentials for heterogeneous catalysiscited by 4
  9. 2025Understanding and Mitigating Distribution Shifts For Machine Learning Force Fieldscited by 3
  10. 2025Equivariant Spherical Transformer for Efficient Molecular Modelingcited by 3
  11. 2026E2Former-V2: On-the-Fly Equivariant Attention with Linear Activation Memorycited by 3
  12. 2026Elign: Equivariant Diffusion Model Alignment from Foundational Machine Learning Force Fieldscited by 3
  13. 2026Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentialscited by 3
  14. 2025AlphaNet: Scaling Up Local-frame-based Atomistic Interatomic Potentialcited by 2
  15. 2026Global Plane Waves From Local Gaussians: Periodic Charge Densities in a Blinkcited by 2
  16. 2025Diagonal Symmetrization of Neural Network Solvers for the Many-Electron Schr"odinger Equationcited by 1
  17. 2025Tensor Decomposition Networks for Fast Machine Learning Interatomic Potential Computationscited by 1
  18. 2025Scaling Laws for Neural Material Modelscited by 1
  19. 2025Training Dynamics of Learning 3D-Rotational Equivariancecited by 1
  20. 2026From Evaluation to Design: Using Potential Energy Surface Smoothness Metrics to Guide Machine Learning Interatomic Potential Architecturescited by 1
  21. 2025Facet: highly efficient E(3)-equivariant networks for interatomic potentialscited by 0
  22. 2025Comparing the latent features of universal machine-learning interatomic potentialscited by 0
  23. 2026Broken neural scaling laws in materials sciencecited by 0
  24. 2026Practical Deep Heteroskedastic Regressioncited by 0
  25. 2026Using large language models to probe the limits of atom-centered structural descriptorscited by 0
  26. 2026Rotational Equivariance in Machine Learning: A Comprehensive Tutorialcited by 0
  27. 2026Ensemble-Conditioned Molecular Designcited by 0
  28. 2026Complete Neural Electronic Initialization Accelerates Materials DFTcited by 0