# The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups

arXiv: https://arxiv.org/abs/2606.20547 · HTML: https://mlsanity.com/paper/2606.20547/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data](https://mlsanity.com/paper/2002.12880/) (2020)
- [LieTransformer: Equivariant self-attention for Lie Groups](https://mlsanity.com/paper/2012.10885/) (2020)
- [GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers](https://mlsanity.com/paper/2310.10375/) (2023)
- [Geometric Algebra Transformer](https://mlsanity.com/paper/2305.18415/) (2023)
- [Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Space](https://mlsanity.com/paper/2310.02970/) (2023)
- [ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matching](https://mlsanity.com/paper/2409.04576/) (2024)
- [LieRE: Lie Rotational Positional Encodings](https://mlsanity.com/paper/2406.10322/) (2024)
- [Euclidean, Projective, Conformal: Choosing a Geometric Algebra for Equivariant Transformers](https://mlsanity.com/paper/2311.04744/) (2023)
- [Lie Group Decompositions for Equivariant Neural Networks](https://mlsanity.com/paper/2310.11366/) (2023)
- [EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers](https://mlsanity.com/paper/2604.09130/) (2026)
- [RiemannFormer: A Framework for Attention in Curved Spaces](https://mlsanity.com/paper/2506.07405/) (2025)
- [Group Representational Position Encoding](https://mlsanity.com/paper/2512.07805/) (2025)
- [Equivariant Neural Networks for General Linear Symmetries on Lie Algebras](https://mlsanity.com/paper/2510.22984/) (2025)
- [Vanilla Group Equivariant Vision Transformer: Simple and Effective](https://mlsanity.com/paper/2602.08047/) (2026)
- [Toward Manifest Relationality in Transformers via Symmetry Reduction](https://mlsanity.com/paper/2602.18948/) (2026)
- [Clebsch-Gordan Transformer: Fast and Global Equivariant Attention](https://mlsanity.com/paper/2509.24093/) (2025)
- [Efficient Equivariant Transformer for Self-Driving Agent Modeling](https://mlsanity.com/paper/2604.01466/) (2026)
- [Exact Symmetry as Algebra: Group-Structured Tensors that Enforce Physical Selection Rules](https://mlsanity.com/paper/2605.20440/) (2026)

## Abstract (arXiv)

We place the attention token on the group: a token is an element gi of a matrix Lie group G -- a bare transformation, with no feature payload and no external action ρ(g) carrying it. To our knowledge this is the first attention construction whose tokens are bare matrix Lie group elements: their score is the closed-form algebra norm of the relative pose rather than a learned kernel, and it reaches the affine full-frame groups that every irrep- or surjective-exp-based method must exclude. We call it Lie-Algebra Attention. Once tokens are group elements, the rest follows with none of the usual representation-theoretic machinery. The relative geometry of a pair is canonical, gi-1 gj, so the pairwise invariant wij = log(gi-1 gj) is intrinsic rather than designed; equivariance under the diagonal G-action is tautological, and the cocycle condition holds automatically. The attention score is the negative squared algebra norm, sij = -|log(gi-1 gj)|λ2/τ: the canonical proximity kernel under a block-weighted Frobenius inner product, with no irreducible representations, spherical harmonics, Clebsch-Gordan products, or learned kernel. The construction applies to any matrix Lie group on a chosen logarithm chart containing the relative poses, including the non-compact non-abelian affine groups with scale and shear that no vector-token attention method reaches: neither the irrep tradition nor surjective-exp methods. Three sequence-completion experiments, on SE(2), SO(3), and Aff(2), bear this out: the closed-form score matches a learned MLP kernel on the same invariant and outperforms it on SE(2), using 50 to 80x fewer score parameters, while a vector-token baseline breaks invariance by five to twelve orders of magnitude.