mlsanityData to 7 Oct 2026

Platonic Transformers: A Solid Choice For Equivariance

arXiv 2510.03511 ↗First posted 3 Oct 2025cs.CV cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through complex, computationally intensive designs. We introduce the Platonic Transformer to resolve this trade-off. By defining attention relative to reference frames from the Platonic solid symmetry groups, our method induces a principled weight-sharing scheme. This enables combined equivariance to continuous translations and Platonic symmetries, while preserving the exact architecture and computational cost of a standard Transformer. Furthermore, we show that this attention is formally equivalent to a dynamic group convolution, which reveals that the model learns adaptive geometric filters and enables a highly scalable, linear-time convolutional variant. Across diverse benchmarks in computer vision (CIFAR-10), 3D point clouds (ScanObjectNN), and molecular property prediction (QM9, OMol25), the Platonic Transformer achieves competitive performance by leveraging these geometric constraints at no additional cost.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

55 of 78 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2020Point Transformercited by 1,462
  11. 2016Group Equivariant Convolutional Networkscited by 1,154
  12. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
38 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021E(n) Equivariant Graph Neural Networkscited by 951
  3. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  6. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  7. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  8. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  9. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  10. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
  11. 2022DeiT III: Revenge of the ViTcited by 417
  12. 2016Steerable CNNscited by 298
  13. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 258
  14. 2020Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Moleculescited by 254
  15. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 225
  16. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
  17. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 201
  18. 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 185
  19. 2022TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentialscited by 179
  20. 2021Spherical Message Passing for 3D Graph Networkscited by 166
  21. 2021Frame Averaging for Invariant and Equivariant Network Designcited by 166
  22. 2023SE(3)-Stochastic Flow Matching for Protein Backbone Generationcited by 147
  23. 2019B-Spline CNNs on Lie Groupscited by 97
  24. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  25. 2025All-atom Diffusion Transformers: Unified generative modelling of molecules and materialscited by 75
  26. 2020Group Equivariant Stand-Alone Self-Attention For Visioncited by 63
  27. 2023Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Spacecited by 40
  28. 2024The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domainscited by 40
  29. 2024Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Productscited by 35
  30. 2022GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Predictioncited by 28
  31. 2022VN-Transformer: Rotation-Equivariant Attention for Vector Neuronscited by 26
  32. 2024Clifford-Steerable Convolutional Neural Networkscited by 24
  33. 2022Equivariant Graph Attention Networks for Molecular Property Predictioncited by 19
  34. 2024Grounding Continuous Representations in Geometry: Equivariant Neural Fieldscited by 19
  35. 2024Space-Time Continuous PDE Forecasting using Equivariant Neural Fieldscited by 18
  36. 2025Improving Equivariant Networks with Probabilistic Symmetry Breakingcited by 15
  37. 2024Euclidean Fast Attention -- Machine Learning Global Atomic Representations at Linear Costcited by 12
  38. 2025Probing Equivariance and Symmetry Breaking in Convolutional Networkscited by 12

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materialscited by 3
  2. 2025A Circular Argument : Does RoPE need to be Equivariant for Vision?cited by 2
  3. 2025Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Powercited by 0
  4. 2025Soft Geometric Inductive Bias for Object Centric Dynamicscited by 0
  5. 2026When Does Equivariance Help? Canonical Alignment in Neural Fluid Surrogatescited by 0
  6. 2026A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of O(2)cited by 0
  7. 2026Rotational Equivariance in Machine Learning: A Comprehensive Tutorialcited by 0
  8. 2026Sample Complexity of Equivariant Reinforcement Learningcited by 0
  9. 2026How Do Transformers Learn to Represent Symmetries?cited by 0