# Group Equivariant Stand-Alone Self-Attention For Vision

arXiv: https://arxiv.org/abs/2010.00977 · HTML: https://mlsanity.com/paper/2010.00977/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 30 of 63)

- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Frame Averaging for Invariant and Equivariant Network Design](https://mlsanity.com/paper/2110.03336/) (2021)
- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [Independent SE(3)-Equivariant Models for End-to-End Rigid Protein Docking](https://mlsanity.com/paper/2111.07786/) (2021)
- [GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers](https://mlsanity.com/paper/2310.10375/) (2023)
- [Is Attention Better Than Matrix Decomposition?](https://mlsanity.com/paper/2109.04553/) (2021)
- [Universal Approximation of Functions on Sets](https://mlsanity.com/paper/2107.01959/) (2021)
- [Learning Partial Equivariances from Data](https://mlsanity.com/paper/2110.10211/) (2021)
- [Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation](https://mlsanity.com/paper/2210.17400/) (2022)
- [Unsupervised Learning of Group Invariant and Equivariant Representations](https://mlsanity.com/paper/2202.07559/) (2022)
- [DISCO: accurate Discrete Scale Convolutions](https://mlsanity.com/paper/2106.02733/) (2021)
- [INO: Invariant Neural Operators for Learning Complex Physical Systems with Momentum Conservation](https://mlsanity.com/paper/2212.14365/) (2022)
- [SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space](https://mlsanity.com/paper/2204.02394/) (2022)
- [E(2)-Equivariant Vision Transformer](https://mlsanity.com/paper/2306.06722/) (2023)
- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification](https://mlsanity.com/paper/2202.07570/) (2022)
- [Unified Fourier-based Kernel and Nonlinearity Design for Equivariant Networks on Homogeneous Spaces](https://mlsanity.com/paper/2206.08362/) (2022)
- [Frame Averaging for Equivariant Shape Space Learning](https://mlsanity.com/paper/2112.01741/) (2021)
- [Equivariance Discovery by Learned Parameter-Sharing](https://mlsanity.com/paper/2204.03640/) (2022)
- [Modelling Long Range Dependencies in ND: From Task-Specific to a General Purpose CNN](https://mlsanity.com/paper/2301.10540/) (2023)
- [Equivariant Networks for Pixelized Spheres](https://mlsanity.com/paper/2106.06662/) (2021)
- [An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models](https://mlsanity.com/paper/2212.14852/) (2022)
- [Equivariant Mesh Attention Networks](https://mlsanity.com/paper/2205.10662/) (2022)
- [Efficient Equivariant Transfer Learning from Pretrained Models](https://mlsanity.com/paper/2305.09900/) (2023)
- [Exploiting Code Symmetries for Learning Program Semantics](https://mlsanity.com/paper/2308.03312/) (2023)
- [Platonic Transformers: A Solid Choice For Equivariance](https://mlsanity.com/paper/2510.03511/) (2025)
- [Task-Equivariant Graph Few-shot Learning](https://mlsanity.com/paper/2305.18758/) (2023)
- [Steerable Transformers for Volumetric Data](https://mlsanity.com/paper/2405.15932/) (2024)
- [Diffusion Generative Modeling on Lie Group Representations](https://mlsanity.com/paper/2502.02513/) (2025)
- [Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration](https://mlsanity.com/paper/2202.03967/) (2022)

## Abstract (arXiv)

We provide a general self-attention formulation to impose group equivariance to arbitrary symmetry groups. This is achieved by defining positional encodings that are invariant to the action of the group considered. Since the group acts on the positional encoding directly, group equivariant self-attention networks (GSA-Nets) are steerable by nature. Our experiments on vision benchmarks demonstrate consistent improvements of GSA-Nets over non-equivariant self-attention networks.