# Quantised Transforming Auto-Encoders: Achieving Equivariance to Arbitrary Transformations in Deep Networks

arXiv: https://arxiv.org/abs/2111.12873 · HTML: https://mlsanity.com/paper/2111.12873/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 24 of 24)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [Deep Convolutional Inverse Graphics Network](https://mlsanity.com/paper/1503.03167/) (2015)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [Transformation-Grounded Image Generation Network for Novel 3D View Synthesis](https://mlsanity.com/paper/1703.02921/) (2017)
- [Multi-view 3D Models from Single Images with a Convolutional Network](https://mlsanity.com/paper/1511.06702/) (2015)
- [Equivariant Neural Rendering](https://mlsanity.com/paper/2006.07630/) (2020)
- [Locally Scale-Invariant Convolutional Neural Networks](https://mlsanity.com/paper/1412.5104/) (2014)
- [Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis](https://mlsanity.com/paper/1601.00706/) (2016)
- [Warped Convolutions: Efficient Invariance to Spatial Transformations](https://mlsanity.com/paper/1609.04382/) (2016)
- [Learning Generalized Transformation Equivariant Representations via Autoencoding Transformations](https://mlsanity.com/paper/1906.08628/) (2019)
- [Novel Views of Objects from a Single Image](https://mlsanity.com/paper/1602.00328/) (2016)
- [Learning Stable Group Invariant Representations with Convolutional Networks](https://mlsanity.com/paper/1301.3537/) (2013)

## Built on by (most cited first, 1 of 1)

- [DUET: 2D Structured and Approximately Equivariant Representations](https://mlsanity.com/paper/2306.16058/) (2023)

## Abstract (arXiv)

In this work we investigate how to achieve equivariance to input transformations in deep networks, purely from data, without being given a model of those transformations. Convolutional Neural Networks (CNNs), for example, are equivariant to image translation, a transformation that can be easily modelled (by shifting the pixels vertically or horizontally). Other transformations, such as out-of-plane rotations, do not admit a simple analytic model. We propose an auto-encoder architecture whose embedding obeys an arbitrary set of equivariance relations simultaneously, such as translation, rotation, colour changes, and many others. This means that it can take an input image, and produce versions transformed by a given amount that were not observed before (e.g. a different point of view of the same object, or a colour variation). Despite extending to many (even non-geometric) transformations, our model reduces exactly to a CNN in the special case of translation-equivariance. Equivariances are important for the interpretability and robustness of deep networks, and we demonstrate results of successful re-rendering of transformed versions of input images on several synthetic and real datasets, as well as results on object pose estimation.