mlsanityData to 7 Oct 2026

EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations

arXiv 2306.12059 ↗First posted 21 Jun 2023cs.LG cs.AI physics.comp-phcited by 201

Not on the map. Why?

Abstract (arXiv)

Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace SO(3) convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements -- attention re-normalization, separable S2 activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to 9% on forces, 4% on energies, offers better speed-accuracy trade-offs, and 2× reduction in DFT calculations needed for computing adsorption energies. Additionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). Builds on 54 of its 80 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

54 of 80 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2017Graph Attention Networkscited by 8,854
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2021E(n) Equivariant Graph Neural Networkscited by 951
38 more
  1. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  2. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  3. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  4. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  6. 2020A Generalization of Transformer Networks to Graphscited by 653
  7. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  8. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  9. 2020Directional Message Passing for Molecular Graphscited by 556
  10. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  11. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 467
  12. 2018Spherical CNNscited by 452
  13. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
  14. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  15. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  16. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 308
  17. 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
  18. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  19. 2020Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Moleculescited by 254
  20. 2022Learning Local Equivariant Representations for Large-Scale Atomistic Dynamicscited by 254
  21. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  22. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 225
  23. 2017Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanicscited by 213
  24. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 179
  25. 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 168
  26. 2021Spherical Message Passing for 3D Graph Networkscited by 166
  27. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  28. 2023Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNscited by 113
  29. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  30. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  31. 2021SE(3)-equivariant prediction of molecular wavefunctions and electronic densitiescited by 67
  32. 2020OrbNet: Deep Learning for Quantum Chemistry Using Symmetry-Adapted Atomic-Orbital Featurescited by 63
  33. 2022Benchmarking Graphormer on Large-Scale Molecular Modeling Datasetscited by 63
  34. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  35. 2021ForceNet: A Graph Neural Network for Large-Scale Quantum Calculationscited by 58
  36. 2021Rotation Invariant Graph Neural Networks using Spin Convolutionscited by 52
  37. 2020Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Propertiescited by 51
  38. 2022Spherical Channels for Modeling Atomic Interactionscited by 47

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 42; 2025: 86; 2026: 64.

201 papers.

Sort:

By area:

  1. 2024FlowMM: Generating Materials with Riemannian Flow Matchingcited by 105
  2. 2025UMA: A Family of Universal Models for Atomscited by 99
  3. 2025Learning Smooth and Expressive Interatomic Potentials for Physical Property Predictioncited by 79
  4. 2025All-atom Diffusion Transformers: Unified generative modelling of molecules and materialscited by 75
  5. 2024Orb: A Fast, Scalable Neural Network Potentialcited by 63
  6. 2024FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributionscited by 52
  7. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 51
  8. 2024Open Materials 2024 (OMat24) Inorganic Materials Dataset and Modelscited by 50
  9. 2024Equivariant Frames and the Impossibility of Continuous Canonicalizationcited by 41
  10. 2024ET-Flow: Equivariant Flow-Matching for Molecular Conformer Generationcited by 41
  11. 2024The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domainscited by 40
  12. 2024Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Productscited by 35
38 more
  1. 2023Matbench Discovery -- A framework to evaluate machine learning crystal stability predictionscited by 34
  2. 2023Crystal-GFN: sampling crystals with desirable properties and constraintscited by 29
  3. 2024A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applicationscited by 28
  4. 2025A practical guide to machine learning interatomic potentials -- Status and futurecited by 24
  5. 2024The dark side of the forces: assessing non-conservative force models for atomistic machine learningcited by 23
  6. 2023Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolutioncited by 19
  7. 2023The Open DAC 2023 Dataset and Challenges for Sorbent Discovery in Direct Air Capturecited by 18
  8. 2024Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fieldscited by 18
  9. 2024Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?cited by 18
  10. 2023Uncertainty-biased molecular dynamics for learning uniformly accurate interatomic potentialscited by 16
  11. 2023Learning to design protein-protein interactions with enhanced generalizationcited by 15
  12. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  13. 2025Flow Matching Meets Biology and Life Science: A Surveycited by 14
  14. 2024Higher-Rank Irreducible Cartesian Tensors for Equivariant Message Passingcited by 13
  15. 2024Beyond Canonicalization: How Tensorial Messages Improve Equivariant Message Passingcited by 13
  16. 2024Relaxed Equivariance via Multitask Learningcited by 13
  17. 2025Enhancing the Scalability and Applicability of Kohn-Sham Hamiltonians for Molecular Systemscited by 13
  18. 2025Probing Equivariance and Symmetry Breaking in Convolutional Networkscited by 12
  19. 2025E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Productscited by 12
  20. 2025A Survey of Graph Transformers: Architectures, Theories and Applicationscited by 12
  21. 2025Universally Converging Representations of Matter Across Scientific Foundation Modelscited by 12
  22. 2025FlashMD: long-stride, universal prediction of molecular dynamicscited by 11
  23. 2025PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Designcited by 11
  24. 2025HIP: Hessian Interatomic Potentials without derivativescited by 11
  25. 2026Pushing the limits of unconstrained machine-learned interatomic potentialscited by 11
  26. 2024AI-driven inverse design of materials: Past, present and futurecited by 10
  27. 2025Efficient and Scalable Density Functional Theory Hamiltonian Prediction through Adaptive Sparsitycited by 10
  28. 2025How simple can you go? An off-the-shelf transformer approach to molecular dynamicscited by 10
  29. 2025ELECTRA: A Cartesian Network for 3D Charge Density Prediction with Floating Orbitalscited by 9
  30. 2025Platonic Transformers: A Solid Choice For Equivariancecited by 9
  31. 2024AdsorbDiff: Adsorbate Placement via Conditional Denoising Diffusioncited by 8
  32. 2024Steerable Transformers for Volumetric Datacited by 8
  33. 2024A Recipe for Charge Density Predictioncited by 8
  34. 2024Adsorb-Agent: Autonomous Identification of Stable Adsorption Configurations via Large Language Model Agentcited by 8
  35. 2024Learning local equivariant representations for quantum operatorscited by 7
  36. 2024Relaxed Equivariant Graph Neural Networkscited by 7
  37. 2025MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Modelscited by 7
  38. 2025SE(3)-Equivariant Diffusion Policy in Spherical Fourier Spacecited by 7

50 most cited of 201 All 201 (JSON)