# Unsupervised Object Representation Learning using Translation and Rotation Group Equivariant VAE

arXiv: https://arxiv.org/abs/2210.12918 · HTML: https://mlsanity.com/paper/2210.12918/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Adversarial Generation of Continuous Images](https://mlsanity.com/paper/2011.12026/) (2020)
- [B-Spline CNNs on Lie Groups](https://mlsanity.com/paper/1909.12057/) (2019)
- [PixelVAE: A Latent Variable Model for Natural Images](https://mlsanity.com/paper/1611.05013/) (2016)
- [Group Equivariant Capsule Networks](https://mlsanity.com/paper/1806.05086/) (2018)
- [Explicitly disentangling image content from translation and rotation with spatial-VAE](https://mlsanity.com/paper/1909.11663/) (2019)
- [Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs](https://mlsanity.com/paper/1803.08207/) (2018)
- [CocoNet: A deep neural network for mapping pixel coordinates to color values](https://mlsanity.com/paper/1805.11357/) (2018)

## Built on by (most cited first, 8 of 8)

- [Tree Variational Autoencoders](https://mlsanity.com/paper/2306.08984/) (2023)
- [MELON: NeRF with Unposed Images in SO(3)](https://mlsanity.com/paper/2303.08096/) (2023)
- [Rotation and Translation Invariant Representation Learning with Implicit Neural Representations](https://mlsanity.com/paper/2304.13995/) (2023)
- [CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational AutoEncoder](https://mlsanity.com/paper/2401.08897/) (2024)
- [Latent Code Augmentation Based on Stable Diffusion for Data-free Substitute Attacks](https://mlsanity.com/paper/2307.12872/) (2023)
- [Equivariant amortized inference of poses for cryo-EM](https://mlsanity.com/paper/2406.01630/) (2024)
- [Joint Denoising of Cryo-EM Projection Images using Polar Transformers](https://mlsanity.com/paper/2506.11283/) (2025)
- [Cryo-SWAN: the Multi-Scale Wavelet-decomposition-inspired Autoencoder Network for molecular density representation of molecular volumes](https://mlsanity.com/paper/2603.03342/) (2026)

## Abstract (arXiv)

In many imaging modalities, objects of interest can occur in a variety of locations and poses (i.e. are subject to translations and rotations in 2d or 3d), but the location and pose of an object does not change its semantics (i.e. the object's essence). That is, the specific location and rotation of an airplane in satellite imagery, or the 3d rotation of a chair in a natural image, or the rotation of a particle in a cryo-electron micrograph, do not change the intrinsic nature of those objects. Here, we consider the problem of learning semantic representations of objects that are invariant to pose and location in a fully unsupervised manner. We address shortcomings in previous approaches to this problem by introducing TARGET-VAE, a translation and rotation group-equivariant variational autoencoder framework. TARGET-VAE combines three core innovations: 1) a rotation and translation group-equivariant encoder architecture, 2) a structurally disentangled distribution over latent rotation, translation, and a rotation-translation-invariant semantic object representation, which are jointly inferred by the approximate inference network, and 3) a spatially equivariant generator network. In comprehensive experiments, we show that TARGET-VAE learns disentangled representations without supervision that significantly improve upon, and avoid the pathologies of, previous methods. When trained on images highly corrupted by rotation and translation, the semantic representations learned by TARGET-VAE are similar to those learned on consistently posed objects, dramatically improving clustering in the semantic latent space. Furthermore, TARGET-VAE is able to perform remarkably accurate unsupervised pose and location inference. We expect methods like TARGET-VAE will underpin future approaches for unsupervised object generation, pose prediction, and object detection.