# Are We Using Autoencoders in a Wrong Way?

arXiv: https://arxiv.org/abs/2309.01532 · HTML: https://mlsanity.com/paper/2309.01532/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Topological Autoencoders](https://mlsanity.com/paper/1906.00722/) (2019)
- [From Principal Subspaces to Principal Components with Linear Autoencoders](https://mlsanity.com/paper/1804.10253/) (2018)
- [Chart Auto-Encoders for Manifold Structured Data](https://mlsanity.com/paper/1912.10094/) (2019)
- [A Survey of Inductive Biases for Factorial Representation-Learning](https://mlsanity.com/paper/1612.05299/) (2016)
- [Extendable and invertible manifold learning with geometry regularized autoencoders](https://mlsanity.com/paper/2007.07142/) (2020)
- [Exploring the Latent Space of Autoencoders with Interventional Assays](https://mlsanity.com/paper/2106.16091/) (2021)
- [Reducing Data Complexity using Autoencoders with Class-informed Loss Functions](https://mlsanity.com/paper/2111.06142/) (2021)
- [A Discussion On the Validity of Manifold Learning](https://mlsanity.com/paper/2106.01608/) (2021)

## Built on by (most cited first, 2 of 2)

- [Sparse L1-Autoencoders for Scientific Data Compression](https://mlsanity.com/paper/2405.14270/) (2024)
- [Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction](https://mlsanity.com/paper/2608.29867/) (2026)

## Abstract (arXiv)

Autoencoders are certainly among the most studied and used Deep Learning models: the idea behind them is to train a model in order to reconstruct the same input data. The peculiarity of these models is to compress the information through a bottleneck, creating what is called Latent Space. Autoencoders are generally used for dimensionality reduction, anomaly detection and feature extraction. These models have been extensively studied and updated, given their high simplicity and power. Examples are (i) the Denoising Autoencoder, where the model is trained to reconstruct an image from a noisy one; (ii) Sparse Autoencoder, where the bottleneck is created by a regularization term in the loss function; (iii) Variational Autoencoder, where the latent space is used to generate new consistent data. In this article, we revisited the standard training for the undercomplete Autoencoder modifying the shape of the latent space without using any explicit regularization term in the loss function. We forced the model to reconstruct not the same observation in input, but another one sampled from the same class distribution. We also explored the behaviour of the latent space in the case of reconstruction of a random sample from the whole dataset.