# VN-Transformer: Rotation-Equivariant Attention for Vector Neurons

arXiv: https://arxiv.org/abs/2206.04176 · HTML: https://mlsanity.com/paper/2206.04176/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Rotation equivariant vector field networks](https://mlsanity.com/paper/1612.09346/) (2016)
- [Approximately Equivariant Networks for Imperfectly Symmetric Dynamics](https://mlsanity.com/paper/2201.11969/) (2022)
- [Polar Transformer Networks](https://mlsanity.com/paper/1709.01889/) (2017)
- [Rotation Invariant Convolutions for 3D Point Clouds Deep Learning](https://mlsanity.com/paper/1908.06297/) (2019)
- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)

## Built on by (most cited first, 27 of 27)

- [EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation](https://mlsanity.com/paper/2310.16050/) (2023)
- [EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction](https://mlsanity.com/paper/2308.06564/) (2023)
- [Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance](https://mlsanity.com/paper/2305.16314/) (2023)
- [EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervision](https://mlsanity.com/paper/2303.15440/) (2023)
- [Regularizing Towards Soft Equivariance Under Mixed Symmetries](https://mlsanity.com/paper/2306.00356/) (2023)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [Single Mesh Diffusion Models with Field Latents for Texture Generation](https://mlsanity.com/paper/2312.09250/) (2023)
- [Attention on the Sphere](https://mlsanity.com/paper/2505.11157/) (2025)
- [Platonic Transformers: A Solid Choice For Equivariance](https://mlsanity.com/paper/2510.03511/) (2025)
- [SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation](https://mlsanity.com/paper/2411.07326/) (2024)
- [Geometric Hyena Networks for Large-scale Equivariant Learning](https://mlsanity.com/paper/2505.22560/) (2025)
- [Equivariant Neural Networks for General Linear Symmetries on Lie Algebras](https://mlsanity.com/paper/2510.22984/) (2025)
- [Quick ViTs: Speeding up Vision Transformers through Equivariance](https://mlsanity.com/paper/2505.15441/) (2025)
- [Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation](https://mlsanity.com/paper/2308.04725/) (2023)
- [Current Symmetry Group Equivariant Convolution Frameworks for Representation Learning](https://mlsanity.com/paper/2409.07327/) (2024)
- [Learning Generalizable Shape Completion with SIM(3) Equivariance](https://mlsanity.com/paper/2509.26631/) (2025)
- [Harmformer: Harmonic Networks Meet Transformers for Continuous Roto-Translation Equivariance](https://mlsanity.com/paper/2411.03794/) (2024)
- [VENI: Variational Encoder for Natural Illumination](https://mlsanity.com/paper/2601.14079/) (2026)
- [Efficient Equivariant Transformer for Self-Driving Agent Modeling](https://mlsanity.com/paper/2604.01466/) (2026)
- [Exact Symmetry as Algebra: Group-Structured Tensors that Enforce Physical Selection Rules](https://mlsanity.com/paper/2605.20440/) (2026)
- [REViT: Roto-reflection Equivariant Convolutional Vision Transformer](https://mlsanity.com/paper/2606.25318/) (2026)
- [E3-Net: Efficient E(3)-Equivariant Normal Estimation Network](https://mlsanity.com/paper/2406.00347/) (2024)
- [Towards Foundation Models for Cryo-ET Subtomogram Analysis](https://mlsanity.com/paper/2509.24311/) (2025)
- [REVNET: Rotation-Equivariant Point Cloud Completion via Vector Neuron Anchor Transformer](https://mlsanity.com/paper/2601.08558/) (2026)
- [On the Role of Rotation Equivariance in Monocular 2D-to-3D Human Pose Lifting](https://mlsanity.com/paper/2601.13913/) (2026)
- [A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of O(2)](https://mlsanity.com/paper/2606.27864/) (2026)
- [E3DGS: Unified Geometric-Photometric Equivariance for 3D Gaussian Splatting via Color-as-Geometry Embedding](https://mlsanity.com/paper/2607.15536/) (2026)

## Abstract (arXiv)

Rotation equivariance is a desirable property in many practical applications such as motion forecasting and 3D perception, where it can offer benefits like sample efficiency, better generalization, and robustness to input perturbations. Vector Neurons (VN) is a recently developed framework offering a simple yet effective approach for deriving rotation-equivariant analogs of standard machine learning operations by extending one-dimensional scalar neurons to three-dimensional "vector neurons." We introduce a novel "VN-Transformer" architecture to address several shortcomings of the current VN models. Our contributions are: (i) we derive a rotation-equivariant attention mechanism which eliminates the need for the heavy feature preprocessing required by the original Vector Neurons models; (ii) we extend the VN framework to support non-spatial attributes, expanding the applicability of these models to real-world datasets; (iii) we derive a rotation-equivariant mechanism for multi-scale reduction of point-cloud resolution, greatly speeding up inference and training; (iv) we show that small tradeoffs in equivariance (ε-approximate equivariance) can be used to obtain large improvements in numerical stability and training robustness on accelerated hardware, and we bound the propagation of equivariance violations in our models. Finally, we apply our VN-Transformer to 3D shape classification and motion forecasting with compelling results.