# E(2)-Equivariant Vision Transformer

arXiv: https://arxiv.org/abs/2306.06722 · HTML: https://mlsanity.com/paper/2306.06722/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Exploring Self-attention for Image Recognition](https://mlsanity.com/paper/2004.13621/) (2020)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data](https://mlsanity.com/paper/2002.12880/) (2020)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Learning Steerable Filters for Rotation Equivariant CNNs](https://mlsanity.com/paper/1711.07289/) (2017)

## Built on by (most cited first, 27 of 27)

- [Rumor Detection with a novel graph neural network approach](https://mlsanity.com/paper/2403.16206/) (2024)
- [Image Captioning in news report scenario](https://mlsanity.com/paper/2403.16209/) (2024)
- [Optimizing PatchCore for Few/many-shot Anomaly Detection](https://mlsanity.com/paper/2307.10792/) (2023)
- [Steerable Transformers for Volumetric Data](https://mlsanity.com/paper/2405.15932/) (2024)
- [Flopping for FLOPs: Leveraging equivariance for computational efficiency](https://mlsanity.com/paper/2502.05169/) (2025)
- [Learning to Play Atari in a World of Tokens](https://mlsanity.com/paper/2406.01361/) (2024)
- [Enhancing Image Restoration Transformer via Adaptive Translation Equivariance](https://mlsanity.com/paper/2506.18520/) (2025)
- [Rotation Equivariant Arbitrary-scale Image Super-Resolution](https://mlsanity.com/paper/2508.05160/) (2025)
- [Group Downsampling with Equivariant Anti-aliasing](https://mlsanity.com/paper/2504.17258/) (2025)
- [Quick ViTs: Speeding up Vision Transformers through Equivariance](https://mlsanity.com/paper/2505.15441/) (2025)
- [Vanilla Group Equivariant Vision Transformer: Simple and Effective](https://mlsanity.com/paper/2602.08047/) (2026)
- [Efficient Rotation Invariance in Deep Neural Networks through Artificial Mental Rotation](https://mlsanity.com/paper/2311.08525/) (2023)
- [CLIPSym: Delving into Symmetry Detection with CLIP](https://mlsanity.com/paper/2508.14197/) (2025)
- [Alias-Free ViT: Fractional Shift Invariance via Linear Attention](https://mlsanity.com/paper/2510.22673/) (2025)
- [Contrastive Learning Via Equivariant Representation](https://mlsanity.com/paper/2406.00262/) (2024)
- [Harmformer: Harmonic Networks Meet Transformers for Continuous Roto-Translation Equivariance](https://mlsanity.com/paper/2411.03794/) (2024)
- [RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry](https://mlsanity.com/paper/2603.01544/) (2026)
- [Rotation Equivariant Mamba for Vision Tasks](https://mlsanity.com/paper/2603.09138/) (2026)
- [Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Control](https://mlsanity.com/paper/2406.06072/) (2024)
- [Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power](https://mlsanity.com/paper/2512.09673/) (2025)
- [Equi-ViT: Rotational Equivariant Vision Transformer for Robust Histopathology Analysis](https://mlsanity.com/paper/2601.09130/) (2026)
- [ViT Registers and Fractal ViT](https://mlsanity.com/paper/2601.15506/) (2026)
- [A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of O(2)](https://mlsanity.com/paper/2606.27864/) (2026)
- [Mirror-Fusion Attention for Reflection-Aware Self-Supervised Representation Learning](https://mlsanity.com/paper/2607.00850/) (2026)
- [Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform](https://mlsanity.com/paper/2607.21271/) (2026)
- [Embedding Rotation Invariance for Provable Multi-Oriented Scene Text Recognition](https://mlsanity.com/paper/2608.10684/) (2026)
- [Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention](https://mlsanity.com/paper/2608.30417/) (2026)

## Abstract (arXiv)

Vision Transformer (ViT) has achieved remarkable performance in computer vision. However, positional encoding in ViT makes it substantially difficult to learn the intrinsic equivariance in data. Initial attempts have been made on designing equivariant ViT but are proved defective in some cases in this paper. To address this issue, we design a Group Equivariant Vision Transformer (GE-ViT) via a novel, effective positional encoding operator. We prove that GE-ViT meets all the theoretical requirements of an equivariant neural network. Comprehensive experiments are conducted on standard benchmark datasets, demonstrating that GE-ViT significantly outperforms non-equivariant self-attention networks. The code is available at https://github.com/ZJUCDSYangKaifan/GEVit.