# Geometric Algebra Transformer

arXiv: https://arxiv.org/abs/2305.18415 · HTML: https://mlsanity.com/paper/2305.18415/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 28 May 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities](https://mlsanity.com/paper/2006.02425/) (2020)
- [Geometric and Physical Quantities Improve E(3) Equivariant Message Passing](https://mlsanity.com/paper/2110.02905/) (2021)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs](https://mlsanity.com/paper/2003.05425/) (2020)
- [Clifford Neural Layers for PDE Modeling](https://mlsanity.com/paper/2209.04934/) (2022)

## Built on by (most cited first, 30 of 48)

- [Iterated Denoising Energy Matching for Sampling from Boltzmann Densities](https://mlsanity.com/paper/2402.06121/) (2024)
- [A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems](https://mlsanity.com/paper/2312.07511/) (2023)
- [Does equivariance matter at scale?](https://mlsanity.com/paper/2410.23179/) (2024)
- [A Lorentz-Equivariant Transformer for All of the LHC](https://mlsanity.com/paper/2411.00446/) (2024)
- [Clifford-Steerable Convolutional Neural Networks](https://mlsanity.com/paper/2402.14730/) (2024)
- [E(n) Equivariant Topological Neural Networks](https://mlsanity.com/paper/2405.15429/) (2024)
- [Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems](https://mlsanity.com/paper/2502.17019/) (2025)
- [Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures](https://mlsanity.com/paper/2407.09468/) (2024)
- [Relaxed Equivariance via Multitask Learning](https://mlsanity.com/paper/2410.17878/) (2024)
- [LaB-GATr: geometric algebra transformers for large biomedical surface and volume meshes](https://mlsanity.com/paper/2403.07536/) (2024)
- [Flopping for FLOPs: Leveraging equivariance for computational efficiency](https://mlsanity.com/paper/2502.05169/) (2025)
- [GAGrasp: Geometric Algebra Diffusion for Dexterous Grasping](https://mlsanity.com/paper/2503.04123/) (2025)
- [CliffordNet: All You Need is Geometric Algebra](https://mlsanity.com/paper/2601.06793/) (2026)
- [Differentiable and Learnable Wireless Simulation with Geometric Transformers](https://mlsanity.com/paper/2406.14995/) (2024)
- [SE(3)-Hyena Operator for Scalable Equivariant Learning](https://mlsanity.com/paper/2407.01049/) (2024)
- [Multivector Neurons: Better and Faster O(n)-Equivariant Clifford Graph Neural Networks](https://mlsanity.com/paper/2406.04052/) (2024)
- [Optimal Equivariant Architectures from the Symmetries of Matrix-Element Likelihoods](https://mlsanity.com/paper/2410.18553/) (2024)
- [A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing](https://mlsanity.com/paper/2410.20516/) (2024)
- [Geometric Hyena Networks for Large-scale Equivariant Learning](https://mlsanity.com/paper/2505.22560/) (2025)
- [Equivariant Neural Networks for General Linear Symmetries on Lie Algebras](https://mlsanity.com/paper/2510.22984/) (2025)
- [Clifford Group Equivariant Diffusion Models for 3D Molecular Generation](https://mlsanity.com/paper/2504.15773/) (2025)
- [Hybrid Diffusion Policies with Projective Geometric Algebra for Efficient Robot Manipulation Learning](https://mlsanity.com/paper/2507.05695/) (2025)
- [Learning Hemodynamic Scalar Fields on Coronary Artery Meshes: A Benchmark of Geometric Deep Learning Models](https://mlsanity.com/paper/2501.09046/) (2025)
- [GLGENN: A Novel Parameter-Light Equivariant Neural Networks Architecture Based on Clifford Geometric Algebras](https://mlsanity.com/paper/2506.09625/) (2025)
- [Versor: A Geometric Sequence Architecture](https://mlsanity.com/paper/2602.10195/) (2026)
- [End-to-end event reconstruction for precision physics at future colliders](https://mlsanity.com/paper/2603.04084/) (2026)
- [Randomized Geometric Algebra Methods for Convex Neural Networks](https://mlsanity.com/paper/2406.02806/) (2024)
- [Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenes](https://mlsanity.com/paper/2408.02275/) (2024)
- [Optimal Symmetries in Binary Classification](https://mlsanity.com/paper/2408.08823/) (2024)
- [Efficient Equivariant Transformer for Self-Driving Agent Modeling](https://mlsanity.com/paper/2604.01466/) (2026)

## Abstract (arXiv)

Problems involving geometric data arise in physics, chemistry, robotics, computer vision, and many other fields. Such data can take numerous forms, for instance points, direction vectors, translations, or rotations, but to date there is no single architecture that can be applied to such a wide variety of geometric types while respecting their symmetries. In this paper we introduce the Geometric Algebra Transformer (GATr), a general-purpose architecture for geometric data. GATr represents inputs, outputs, and hidden states in the projective geometric (or Clifford) algebra, which offers an efficient 16-dimensional vector-space representation of common geometric objects as well as operators acting on them. GATr is equivariant with respect to E(3), the symmetry group of 3D Euclidean space. As a Transformer, GATr is versatile, efficient, and scalable. We demonstrate GATr in problems from n-body modeling to wall-shear-stress estimation on large arterial meshes to robotic motion planning. GATr consistently outperforms both non-geometric and equivariant baselines in terms of error, data efficiency, and scalability.