# Understanding Autoencoders with Information Theoretic Concepts

arXiv: https://arxiv.org/abs/1804.00057 · HTML: https://mlsanity.com/paper/1804.00057/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2018.

## Built on by (most cited first, 29 of 29)

- [Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges](https://mlsanity.com/paper/2103.04505/) (2021)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [Understanding Neural Networks with Logarithm Determinant Entropy Estimator](https://mlsanity.com/paper/2105.03705/) (2021)
- [Exploring the Latent Space of Autoencoders with Interventional Assays](https://mlsanity.com/paper/2106.16091/) (2021)
- [Information flows of diverse autoencoders](https://mlsanity.com/paper/2102.07402/) (2021)
- [Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysis](https://mlsanity.com/paper/2207.10939/) (2022)
- [Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning](https://mlsanity.com/paper/2209.12831/) (2022)
- [Pain level and pain-related behaviour classification using GRU-based sparsely-connected RNNs](https://mlsanity.com/paper/2212.14806/) (2022)
- [Training Invertible Neural Networks as Autoencoders](https://mlsanity.com/paper/2303.11239/) (2023)
- [Improving Knot Prediction in Wood Logs with Longitudinal Feature Propagation](https://mlsanity.com/paper/2308.11291/) (2023)
- [Target Detection and Segmentation in Circular-Scan Synthetic-Aperture-Sonar Images using Semi-Supervised Convolutional Encoder-Decoders](https://mlsanity.com/paper/2101.03603/) (2021)
- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [EA: An Event Autoencoder for High-Speed Vision Sensing](https://mlsanity.com/paper/2507.06459/) (2025)
- [Opening the Black Box of Deep Neural Networks in Physical Layer Communication](https://mlsanity.com/paper/2106.01124/) (2021)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Local Intrinsic Dimensional Entropy](https://mlsanity.com/paper/2304.02223/) (2023)
- [Theoretical Analysis of Deep Neural Networks in Physical Layer Communication](https://mlsanity.com/paper/2202.09954/) (2022)
- [Self-Adaptive, Dynamic, Integrated Statistical and Information Theory Learning](https://mlsanity.com/paper/2211.11491/) (2022)
- [Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study](https://mlsanity.com/paper/2307.08572/) (2023)
- [Enhancing Neural Network Interpretability Through Conductance-Based Information Plane Analysis](https://mlsanity.com/paper/2408.14681/) (2024)
- [Understanding Variational Autoencoders with Intrinsic Dimension and Information Imbalance](https://mlsanity.com/paper/2411.01978/) (2024)
- [Understanding the role of autoencoders for stiff dynamical systems using information theory](https://mlsanity.com/paper/2503.06325/) (2025)
- [Mutual information estimation for graph convolutional neural networks](https://mlsanity.com/paper/2203.16887/) (2022)
- [DeepProphet2 -- A Deep Learning Gene Recommendation Engine](https://mlsanity.com/paper/2208.01918/) (2022)
- [On the Adversarial Robustness of Generative Autoencoders in the Latent Space](https://mlsanity.com/paper/2307.02202/) (2023)
- [MOLE: MOdular Learning FramEwork via Mutual Information Maximization](https://mlsanity.com/paper/2308.07772/) (2023)
- [A Simple and Effective Method for Uncertainty Quantification and OOD Detection](https://mlsanity.com/paper/2508.00754/) (2025)
- [Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition](https://mlsanity.com/paper/2510.27651/) (2025)
- [Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models](https://mlsanity.com/paper/2605.15965/) (2026)

## Abstract (arXiv)

Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep learning architectures that resembles a communication channel. By generalizing the information plane to any cost function, and inspecting the roles and dynamics of different layers using layer-wise information quantities, we emphasize the role that mutual information plays in quantifying learning from data. We further suggest and also experimentally validate, for mean square error training, three fundamental properties regarding the layer-wise flow of information and intrinsic dimensionality of the bottleneck layer, using respectively the data processing inequality and the identification of a bifurcation point in the information plane that is controlled by the given data. Our observations have a direct impact on the optimal design of autoencoders, the design of alternative feedforward training methods, and even in the problem of generalization.