# AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data

arXiv: https://arxiv.org/abs/1901.04596 · HTML: https://mlsanity.com/paper/1901.04596/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2019.

## Built on by (most cited first, 30 of 77)

- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [SEED: Self-supervised Distillation For Visual Representation](https://mlsanity.com/paper/2101.04731/) (2021)
- [What to Hide from Your Students: Attention-Guided Masked Image Modeling](https://mlsanity.com/paper/2203.12719/) (2022)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Contrastive Learning with Stronger Augmentations](https://mlsanity.com/paper/2104.07713/) (2021)
- [Self-Supervised Learning of Object Parts for Semantic Segmentation](https://mlsanity.com/paper/2204.13101/) (2022)
- [Divide and Contrast: Self-supervised Learning from Uncurated Data](https://mlsanity.com/paper/2105.08054/) (2021)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [High Fidelity Visualization of What Your Self-Supervised Representation Knows About](https://mlsanity.com/paper/2112.09164/) (2021)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection](https://mlsanity.com/paper/2205.03346/) (2022)
- [Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2206.06363/) (2022)
- [Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning](https://mlsanity.com/paper/2107.01886/) (2021)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [Improving Transferability of Representations via Augmentation-Aware Self-Supervision](https://mlsanity.com/paper/2111.09613/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images](https://mlsanity.com/paper/2405.20469/) (2024)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Geographical Knowledge-driven Representation Learning for Remote Sensing Images](https://mlsanity.com/paper/2107.05276/) (2021)
- [Max-Margin Contrastive Learning](https://mlsanity.com/paper/2112.11450/) (2021)
- [Self-supervised Learning for Pre-Training 3D Point Clouds: A Survey](https://mlsanity.com/paper/2305.04691/) (2023)
- [Adaptive Contrast for Image Regression in Computer-Aided Disease Assessment](https://mlsanity.com/paper/2112.11700/) (2021)
- [Semi-Supervised Deep Regression with Uncertainty Consistency and Variational Model Ensembling via Bayesian Neural Networks](https://mlsanity.com/paper/2302.07579/) (2023)
- [Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization](https://mlsanity.com/paper/2103.04813/) (2021)
- [GLASS: Guided Latent Slot Diffusion for Object-Centric Learning](https://mlsanity.com/paper/2407.17929/) (2024)
- [Masked Autoencoders are Robust Data Augmentors](https://mlsanity.com/paper/2206.04846/) (2022)
- [Semantic-Aware Generation for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2111.13163/) (2021)
- [Correlational Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2303.12670/) (2023)

## Abstract (arXiv)

The success of deep neural networks often relies on a large amount of labeled examples, which can be difficult to obtain in many real scenarios. To address this challenge, unsupervised methods are strongly preferred for training neural networks without using any labeled data. In this paper, we present a novel paradigm of unsupervised representation learning by Auto-Encoding Transformation (AET) in contrast to the conventional Auto-Encoding Data (AED) approach. Given a randomly sampled transformation, AET seeks to predict it merely from the encoded features as accurately as possible at the output end. The idea is the following: as long as the unsupervised features successfully encode the essential information about the visual structures of original and transformed images, the transformation can be well predicted. We will show that this AET paradigm allows us to instantiate a large variety of transformations, from parameterized, to non-parameterized and GAN-induced ones. Our experiments show that AET greatly improves over existing unsupervised approaches, setting new state-of-the-art performances being greatly closer to the upper bounds by their fully supervised counterparts on CIFAR-10, ImageNet and Places datasets.