# VNT-Net: Rotational Invariant Vector Neuron Transformers

arXiv: https://arxiv.org/abs/2205.09690 · HTML: https://mlsanity.com/paper/2205.09690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Exploring Self-attention for Image Recognition](https://mlsanity.com/paper/2004.13621/) (2020)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)
- [Point Convolutional Neural Networks by Extension Operators](https://mlsanity.com/paper/1803.10091/) (2018)
- [Modeling Point Clouds with Self-Attention and Gumbel Subset Sampling](https://mlsanity.com/paper/1904.03375/) (2019)
- [ShellNet: Efficient Point Cloud Convolutional Neural Networks using Concentric Shells Statistics](https://mlsanity.com/paper/1908.06295/) (2019)
- [Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network](https://mlsanity.com/paper/1811.02565/) (2018)
- [Rotation Invariant Convolutions for 3D Point Clouds Deep Learning](https://mlsanity.com/paper/1908.06297/) (2019)
- [On the Universality of Rotation Equivariant Point Cloud Networks](https://mlsanity.com/paper/2010.02449/) (2020)

## Abstract (arXiv)

Learning 3D point sets with rotational invariance is an important and challenging problem in machine learning. Through rotational invariant architectures, 3D point cloud neural networks are relieved from requiring a canonical global pose and from exhaustive data augmentation with all possible rotations. In this work, we introduce a rotational invariant neural network by combining recently introduced vector neurons with self-attention layers to build a point cloud vector neuron transformer network (VNT-Net). Vector neurons are known for their simplicity and versatility in representing SO(3) actions and are thereby incorporated in common neural operations. Similarly, Transformer architectures have gained popularity and recently were shown successful for images by applying directly on sequences of image patches and achieving superior performance and convergence. In order to benefit from both worlds, we combine the two structures by mainly showing how to adapt the multi-headed attention layers to comply with vector neurons operations. Through this adaptation attention layers become SO(3) and the overall network becomes rotational invariant. Experiments demonstrate that our network efficiently handles 3D point cloud objects in arbitrary poses. We also show that our network achieves higher accuracy when compared to related state-of-the-art methods and requires less training due to a smaller number of hyperparameters in common classification and segmentation tasks.