# Training or Architecture? How to Incorporate Invariance in Neural Networks

arXiv: https://arxiv.org/abs/2106.10044 · HTML: https://mlsanity.com/paper/2106.10044/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)
- [PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors](https://mlsanity.com/paper/1808.10322/) (2018)
- [Deep Functional Maps: Structured Prediction for Dense Shape Correspondence](https://mlsanity.com/paper/1704.08686/) (2017)
- [3D Point Capsule Networks](https://mlsanity.com/paper/1812.10775/) (2018)
- [Rotation equivariant vector field networks](https://mlsanity.com/paper/1612.09346/) (2016)
- [Polar Transformer Networks](https://mlsanity.com/paper/1709.01889/) (2017)
- [Exploring the Landscape of Spatial Robustness](https://mlsanity.com/paper/1712.02779/) (2017)
- [Deep Shells: Unsupervised Shape Correspondence with Optimal Transport](https://mlsanity.com/paper/2010.15261/) (2020)
- [TI-POOLING: transformation-invariant pooling for feature learning in Convolutional Neural Networks](https://mlsanity.com/paper/1604.06318/) (2016)
- [A Graph-CNN for 3D Point Cloud Classification](https://mlsanity.com/paper/1812.01711/) (2018)
- [Equivariant Transformer Networks](https://mlsanity.com/paper/1901.11399/) (2019)
- [Learning rotation invariant convolutional filters for texture classification](https://mlsanity.com/paper/1604.06720/) (2016)
- [Warped Convolutions: Efficient Invariance to Spatial Transformations](https://mlsanity.com/paper/1609.04382/) (2016)
- [Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness](https://mlsanity.com/paper/1906.11235/) (2019)
- [OperatorNet: Recovering 3D Shapes From Difference Operators](https://mlsanity.com/paper/1904.10754/) (2019)

## Built on by (most cited first, 9 of 9)

- [DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10758/) (2022)
- [How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization](https://mlsanity.com/paper/2210.06441/) (2022)
- [Equivariance versus Augmentation for Spherical Images](https://mlsanity.com/paper/2202.03990/) (2022)
- [From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication](https://mlsanity.com/paper/2310.01211/) (2023)
- [Optimization Dynamics of Equivariant and Augmented Neural Networks](https://mlsanity.com/paper/2303.13458/) (2023)
- [Ensembles provably learn equivariance through data augmentation](https://mlsanity.com/paper/2410.01452/) (2024)
- [Training Dynamics of Learning 3D-Rotational Equivariance](https://mlsanity.com/paper/2512.02303/) (2025)
- [Subspace-Configurable Networks](https://mlsanity.com/paper/2305.13536/) (2023)
- [Equivariance and Augmentation for Bayesian Neural Networks](https://mlsanity.com/paper/2606.26273/) (2026)

## Abstract (arXiv)

Many applications require the robustness, or ideally the invariance, of a neural network to certain transformations of input data. Most commonly, this requirement is addressed by either augmenting the training data, using adversarial training, or defining network architectures that include the desired invariance automatically. Unfortunately, the latter often relies on the ability to enlist all possible transformations, which make such approaches largely infeasible for infinite sets of transformations, such as arbitrary rotations or scaling. In this work, we propose a method for provably invariant network architectures with respect to group actions by choosing one element from a (possibly continuous) orbit based on a fixed criterion. In a nutshell, we intend to 'undo' any possible transformation before feeding the data into the actual network. We analyze properties of such approaches, extend them to equivariant networks, and demonstrate their advantages in terms of robustness as well as computational efficiency in several numerical examples. In particular, we investigate the robustness with respect to rotations of images (which can possibly hold up to discretization artifacts only) as well as the provable rotational and scaling invariance of 3D point cloud classification.