# Maximally Informative Hierarchical Representations of High-Dimensional Data

arXiv: https://arxiv.org/abs/1410.7404 · HTML: https://mlsanity.com/paper/1410.7404/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2014.

## Built on by (most cited first, 4 of 4)

- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs](https://mlsanity.com/paper/2205.01940/) (2022)
- [Exploring higher-order neural network node interactions with total correlation](https://mlsanity.com/paper/2402.04440/) (2024)
- [CALM: A Causal Analysis Language Model for Tabular Data in Complex Systems with Local Scores, Conditional Independence Tests, and Relation Attributes](https://mlsanity.com/paper/2510.09846/) (2025)

## Abstract (arXiv)

We consider a set of probabilistic functions of some input variables as a representation of the inputs. We present bounds on how informative a representation is about input data. We extend these bounds to hierarchical representations so that we can quantify the contribution of each layer towards capturing the information in the original data. The special form of these bounds leads to a simple, bottom-up optimization procedure to construct hierarchical representations that are also maximally informative about the data. This optimization has linear computational complexity and constant sample complexity in the number of variables. These results establish a new approach to unsupervised learning of deep representations that is both principled and practical. We demonstrate the usefulness of the approach on both synthetic and real-world data.