# Information flows of diverse autoencoders

arXiv: https://arxiv.org/abs/2102.07402 · HTML: https://mlsanity.com/paper/2102.07402/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 15 of 15)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Estimating Mixture Entropy with Pairwise Distances](https://mlsanity.com/paper/1706.02419/) (2017)
- [Distributed Variational Representation Learning](https://mlsanity.com/paper/1807.04193/) (2018)
- [Understanding Autoencoders with Information Theoretic Concepts](https://mlsanity.com/paper/1804.00057/) (2018)
- [On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views](https://mlsanity.com/paper/2002.00008/) (2020)
- [Adaptive Estimators Show Information Compression in Deep Neural Networks](https://mlsanity.com/paper/1902.09037/) (2019)
- [Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels](https://mlsanity.com/paper/1909.11396/) (2019)
- [On the Information Plane of Autoencoders](https://mlsanity.com/paper/2005.07783/) (2020)
- [Multivariate Extension of Matrix-based Renyi's α-order Entropy Functional](https://mlsanity.com/paper/1808.07912/) (2018)

## Built on by (most cited first, 6 of 6)

- [Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning](https://mlsanity.com/paper/2209.12831/) (2022)
- [How good are variational autoencoders at transfer learning?](https://mlsanity.com/paper/2304.10767/) (2023)
- [Large Language Model Pruning](https://mlsanity.com/paper/2406.00030/) (2024)
- [An AI-Guided Data Centric Strategy to Detect and Mitigate Biases in Healthcare Datasets](https://mlsanity.com/paper/2311.03425/) (2023)
- [Joint Learning for Scattered Point Cloud Understanding with Hierarchical Self-Distillation](https://mlsanity.com/paper/2312.16902/) (2023)
- [Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models](https://mlsanity.com/paper/2605.15965/) (2026)

## Abstract (arXiv)

The outstanding performance of deep learning in various fields has been a fundamental query, which can be potentially examined using information theory that interprets the learning process as the transmission and compression of information. Information plane analyses of the mutual information between the input-hidden-output layers demonstrated two distinct learning phases of fitting and compression. It is debatable if the compression phase is necessary to generalize the input-output relations extracted from training data. In this study, we investigated this through experiments with various species of autoencoders and evaluated their information processing phase with an accurate kernel-based estimator of mutual information. Given sufficient training data, vanilla autoencoders demonstrated the compression phase, which was amplified after imposing sparsity regularization for hidden activities. However, we found that the compression phase is not universally observed in different species of autoencoders, including variational autoencoders, that have special constraints on network weights or manifold of hidden space. These types of autoencoders exhibited perfect generalization ability for test data without requiring the compression phase. Thus, we conclude that the compression phase is not necessary for generalization in representation learning.