# Information-Ordered Bottlenecks for Adaptive Semantic Compression

arXiv: https://arxiv.org/abs/2305.11213 · HTML: https://mlsanity.com/paper/2305.11213/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [Medical image denoising using convolutional denoising autoencoders](https://mlsanity.com/paper/1608.04667/) (2016)
- [Scikit-dimension: a Python package for intrinsic dimension estimation](https://mlsanity.com/paper/2109.02596/) (2021)
- [PCAAE: Principal Component Analysis Autoencoder for organising the latent space of generative networks](https://mlsanity.com/paper/2006.07827/) (2020)
- [Ordering Dimensions with Nested Dropout Normalizing Flows](https://mlsanity.com/paper/2006.08777/) (2020)
- [Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks](https://mlsanity.com/paper/2012.13762/) (2020)
- [Triangular Dropout: Variable Network Width without Retraining](https://mlsanity.com/paper/2205.01235/) (2022)
- [Learning Compact Convolutional Neural Networks with Nested Dropout](https://mlsanity.com/paper/1412.7155/) (2014)

## Built on by (most cited first, 3 of 3)

- ["Principal Components" Enable A New Language of Images](https://mlsanity.com/paper/2503.08685/) (2025)
- [Distributional Principal Autoencoders](https://mlsanity.com/paper/2404.13649/) (2024)
- [Transfer Learning Beyond the Standard Model](https://mlsanity.com/paper/2510.19168/) (2025)

## Abstract (arXiv)

We present the information-ordered bottleneck (IOB), a neural layer designed to adaptively compress data into latent variables ordered by likelihood maximization. Without retraining, IOB nodes can be truncated at any bottleneck width, capturing the most crucial information in the first latent variables. Unifying several previous approaches, we show that IOBs achieve near-optimal compression for a given encoding architecture and can assign ordering to latent signals in a manner that is semantically meaningful. IOBs demonstrate a remarkable ability to compress embeddings of image and text data, leveraging the performance of SOTA architectures such as CNNs, transformers, and diffusion models. Moreover, we introduce a novel theory for estimating global intrinsic dimensionality with IOBs and show that they recover SOTA dimensionality estimates for complex synthetic data. Furthermore, we showcase the utility of these models for exploratory analysis through applications on heterogeneous datasets, enabling computer-aided discovery of dataset complexity.