mlsanityData to 14 Oct 2026

EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

arXiv 2604.09130 ↗First posted 10 Apr 2026cs.LG cs.AI physics.comp-phcited by 11

Not on the map. Why?

Abstract (arXiv)

As SE(3)-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the SE(3)-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving 1.75× speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-S2 activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling S2 grids. Together, SwiGLU-S2 activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

54 of 82 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Layer Normalizationcited by 5,669
  4. 2018How Powerful are Graph Neural Networks?cited by 4,393
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2016Deep Networks with Stochastic Depthcited by 979
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  10. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  11. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  12. 2021How Attentive are Graph Attention Networks?cited by 835
38 more
  1. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  2. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
  3. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  4. 2020Directional Message Passing for Molecular Graphscited by 561
  5. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 474
  6. 2018Spherical CNNscited by 456
  7. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 441
  8. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 354
  9. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 313
  10. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  11. 2022Learning Local Equivariant Representations for Large-Scale Atomistic Dynamicscited by 256
  12. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  13. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 228
  14. 2017Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanicscited by 214
  15. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 204
  16. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 182
  17. 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 171
  18. 2023Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNscited by 117
  19. 2025UMA: A Family of Universal Models for Atomscited by 105
  20. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  21. 2023On the Expressive Power of Geometric Graph Neural Networkscited by 90
  22. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  23. 2025Learning Smooth and Expressive Interatomic Potentials for Physical Property Predictioncited by 84
  24. 2021SE(3)-equivariant prediction of molecular wavefunctions and electronic densitiescited by 67
  25. 2024Orb: A Fast, Scalable Neural Network Potentialcited by 66
  26. 2024Open Materials 2024 (OMat24) Inorganic Materials Dataset and Modelscited by 52
  27. 2020Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Propertiescited by 51
  28. 2022Spherical Channels for Modeling Atomic Interactionscited by 47
  29. 2024Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Productscited by 36
  30. 2023Matbench Discovery -- A framework to evaluate machine learning crystal stability predictionscited by 35
  31. 2023Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic sizecited by 33
  32. 2024The dark side of the forces: assessing non-conservative force models for atomistic machine learningcited by 24
  33. 2025An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networkscited by 22
  34. 2022Equivariant Graph Attention Networks for Molecular Property Predictioncited by 20
  35. 2023The Open DAC 2023 Dataset and Challenges for Sorbent Discovery in Direct Air Capturecited by 18
  36. 2024Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fieldscited by 18
  37. 2025High-performance training and inference for deep equivariant interatomic potentialscited by 17
  38. 2025The Price of Freedom: Exploring Expressivity and Runtime Tradeoffs in Equivariant Tensor Productscited by 15

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Edge Cluster Expansion with Radial Rotary Attention for Interatomic Potentialscited by 2
  2. 2026Implicit Machine Learning Force Fields Accelerate Molecular Dynamics Simulationscited by 1
  3. 2026The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groupscited by 0
  4. 2026EquiFiLM: Charge-Conditioned Equivariant Force Fields via Feature-wise Linear Modulationcited by 0
  5. 2026E3J: An Efficient and Open-Source Backend for Euclidean Equivariant Operations on GPU and TPUcited by 0
  6. 2026A strategic roadmap for an atomistic machine-learning ecosystemcited by 0
  7. 2026Equivariant Flow Matching for Electron Density Predictioncited by 0
  8. 2026FlashCart: Fast Cartesian Tensor Products for Equivariant Interatomic Potentialscited by 0
  9. 2026Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systemscited by 0
  10. 2026How Do Transformers Learn to Represent Symmetries?cited by 0
  11. 2026On Making SE(2)-Invariant Networks Optimalcited by 0