# Information in Infinite Ensembles of Infinitely-Wide Neural Networks

arXiv: https://arxiv.org/abs/1911.09189 · HTML: https://mlsanity.com/paper/1911.09189/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 14 of 14)

- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Reverse Engineering Self-Supervised Learning](https://mlsanity.com/paper/2305.15614/) (2023)
- [Scene Synthesis via Uncertainty-Driven Attribute Synchronization](https://mlsanity.com/paper/2108.13499/) (2021)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2303.00633/) (2023)
- [Estimating informativeness of samples with Smooth Unique Information](https://mlsanity.com/paper/2101.06640/) (2021)
- [Information Bottleneck: Exact Analysis of (Quantized) Neural Networks](https://mlsanity.com/paper/2106.12912/) (2021)
- [What Do We Maximize in Self-Supervised Learning?](https://mlsanity.com/paper/2207.10081/) (2022)
- [Variance-Covariance Regularization Improves Representation Learning](https://mlsanity.com/paper/2306.13292/) (2023)
- [Machine Learning and Deep Learning -- A review for Ecologists](https://mlsanity.com/paper/2204.05023/) (2022)
- [Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling](https://mlsanity.com/paper/2206.07673/) (2022)
- [Monitoring Shortcut Learning using Mutual Information](https://mlsanity.com/paper/2206.13034/) (2022)
- [Bounding generalization error with input compression: An empirical study with infinite-width networks](https://mlsanity.com/paper/2207.09408/) (2022)
- [Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training](https://mlsanity.com/paper/2312.00105/) (2023)

## Abstract (arXiv)

In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.