# EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

arXiv: https://arxiv.org/abs/2604.09130 · HTML: https://mlsanity.com/paper/2604.09130/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 82 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks](https://mlsanity.com/paper/1810.02244/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics](https://mlsanity.com/paper/2204.05249/) (2022)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Geometric and Physical Quantities Improve E(3) Equivariant Message Passing](https://mlsanity.com/paper/2110.02905/) (2021)
- [Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics](https://mlsanity.com/paper/1707.09571/) (2017)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials](https://mlsanity.com/paper/2202.02541/) (2022)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)
- [Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs](https://mlsanity.com/paper/2302.03655/) (2023)

## Built on by (most cited first, 11 of 11)

- [Edge Cluster Expansion with Radial Rotary Attention for Interatomic Potentials](https://mlsanity.com/paper/2607.10664/) (2026)
- [Implicit Machine Learning Force Fields Accelerate Molecular Dynamics Simulations](https://mlsanity.com/paper/2607.29158/) (2026)
- [The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups](https://mlsanity.com/paper/2606.20547/) (2026)
- [EquiFiLM: Charge-Conditioned Equivariant Force Fields via Feature-wise Linear Modulation](https://mlsanity.com/paper/2607.05559/) (2026)
- [E3J: An Efficient and Open-Source Backend for Euclidean Equivariant Operations on GPU and TPU](https://mlsanity.com/paper/2609.35099/) (2026)
- [A strategic roadmap for an atomistic machine-learning ecosystem](https://mlsanity.com/paper/2609.39090/) (2026)
- [Equivariant Flow Matching for Electron Density Prediction](https://mlsanity.com/paper/2610.02651/) (2026)
- [FlashCart: Fast Cartesian Tensor Products for Equivariant Interatomic Potentials](https://mlsanity.com/paper/2610.06409/) (2026)
- [Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems](https://mlsanity.com/paper/2610.08400/) (2026)
- [How Do Transformers Learn to Represent Symmetries?](https://mlsanity.com/paper/2610.10305/) (2026)
- [On Making SE(2)-Invariant Networks Optimal](https://mlsanity.com/paper/2610.17424/) (2026)

## Abstract (arXiv)

As SE(3)-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the SE(3)-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving 1.75× speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-S2 activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling S2 grids. Together, SwiGLU-S2 activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.