# Reducing Redundancy in the Bottleneck Representation of the Autoencoders

arXiv: https://arxiv.org/abs/2202.04629 · HTML: https://mlsanity.com/paper/2202.04629/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 65 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [Soft Weight-Sharing for Neural Network Compression](https://mlsanity.com/paper/1702.04008/) (2017)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [Medical image denoising using convolutional denoising autoencoders](https://mlsanity.com/paper/1608.04667/) (2016)
- [End-to-end optimization of nonlinear transform codes for perceptual quality](https://mlsanity.com/paper/1607.05006/) (2016)
- [Learning End-to-End Lossy Image Compression: A Benchmark](https://mlsanity.com/paper/2002.03711/) (2020)
- [Deep Convolutional AutoEncoder-based Lossy Image Compression](https://mlsanity.com/paper/1804.09535/) (2018)
- [Robust Anomaly Detection in Images using Adversarial Autoencoders](https://mlsanity.com/paper/1901.06355/) (2019)
- [Towards Conceptual Compression](https://mlsanity.com/paper/1604.08772/) (2016)
- [Learning Better Lossless Compression Using Lossy Compression](https://mlsanity.com/paper/2003.10184/) (2020)
- [Feedback Recurrent Autoencoder for Video Compression](https://mlsanity.com/paper/2004.04342/) (2020)
- [Deep Generative Video Compression](https://mlsanity.com/paper/1810.02845/) (2018)
- [Leveraging Filter Correlations for Deep Model Compression](https://mlsanity.com/paper/1811.10559/) (2018)
- [An End-to-End Compression Framework Based on Convolutional Neural Networks](https://mlsanity.com/paper/1708.00838/) (2017)
- [Auto-encoders: reconstruction versus compression](https://mlsanity.com/paper/1403.7752/) (2014)
- [Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces](https://mlsanity.com/paper/1905.07478/) (2019)
- [Graph Embedding with Data Uncertainty](https://mlsanity.com/paper/2009.00505/) (2020)

## Built on by (most cited first, 5 of 5)

- [WLD-Reg: A Data-dependent Within-layer Diversity Regularizer](https://mlsanity.com/paper/2301.01352/) (2023)
- [Channel-wise Feature Decorrelation for Enhanced Learned Image Compression](https://mlsanity.com/paper/2403.10936/) (2024)
- [Convolutional autoencoder-based multimodal one-class classification](https://mlsanity.com/paper/2309.14090/) (2023)
- [Efficient CNN with uncorrelated Bag of Features pooling](https://mlsanity.com/paper/2209.10865/) (2022)
- [Learning a Sampling-Free Variational DNN Plugin from Tiny Training Sets to Refine OOD Segmentation With Uncertainty Estimation](https://mlsanity.com/paper/2606.15837/) (2026)

## Abstract (arXiv)

Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks, e.g., dimensionality reduction, image compression, and image denoising. An AE has two goals: (i) compress the original input to a low-dimensional space at the bottleneck of the network topology using an encoder, (ii) reconstruct the input from the representation at the bottleneck using a decoder. Both encoder and decoder are optimized jointly by minimizing a distortion-based loss which implicitly forces the model to keep only those variations of input data that are required to reconstruct the and to reduce redundancies. In this paper, we propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. To this end, we propose an additional loss term, based on the pair-wise correlation of the neurons, which complements the standard reconstruction loss forcing the encoder to learn a more diverse and richer representation of the input. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST. The experimental results show that the proposed loss leads consistently to superior performance compared to the standard AE loss.