Information in Infinite Ensembles of Infinitely-Wide Neural Networks
Not on the map. Why?
Abstract (arXiv)
In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Nov 2019.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 5; 2023: 5.
14 papers.
Sort:
- 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
- 2023Reverse Engineering Self-Supervised Learningcited by 33
- 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronizationcited by 29
- 2021PAC-Bayes Information Bottleneckcited by 28
- 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
- 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
- 2022What Do We Maximize in Self-Supervised Learning?cited by 15
- 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 14
- 2023Variance-Covariance Regularization Improves Representation Learningcited by 13
- 2022Machine Learning and Deep Learning -- A review for Ecologistscited by 11
- 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
- 2022Monitoring Shortcut Learning using Mutual Informationcited by 6
- 2023Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training30 Nov 2023
- 2023Variance-Covariance Regularization Improves Representation Learning23 Jun 2023
- 2023Reverse Engineering Self-Supervised Learning24 May 2023
- 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review19 Apr 2023
- 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization1 Mar 2023
- 2022What Do We Maximize in Self-Supervised Learning?20 Jul 2022
- 2022Bounding generalization error with input compression: An empirical study with infinite-width networks19 Jul 2022
- 2022Monitoring Shortcut Learning using Mutual Information27 Jun 2022
- 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling15 Jun 2022
- 2022Machine Learning and Deep Learning -- A review for Ecologists11 Apr 2022
- 2021PAC-Bayes Information Bottleneck29 Sep 2021
- 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronization30 Aug 2021
2 more
- 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
- 2023Reverse Engineering Self-Supervised Learningcited by 33
- 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronizationcited by 29
- 2021PAC-Bayes Information Bottleneckcited by 28
- 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
- 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
- 2022What Do We Maximize in Self-Supervised Learning?cited by 15
- 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networkscited by 14
- 2023Variance-Covariance Regularization Improves Representation Learningcited by 13
- 2022Machine Learning and Deep Learning -- A review for Ecologistscited by 11
- 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingcited by 9
- 2022Monitoring Shortcut Learning using Mutual Informationcited by 6
12 most cited of 14 All 14 (JSON)
- 2023Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training30 Nov 2023
- 2023Variance-Covariance Regularization Improves Representation Learning23 Jun 2023
- 2023Reverse Engineering Self-Supervised Learning24 May 2023
- 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review19 Apr 2023
- 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization1 Mar 2023
- 2022What Do We Maximize in Self-Supervised Learning?20 Jul 2022
- 2022Bounding generalization error with input compression: An empirical study with infinite-width networks19 Jul 2022
- 2022Monitoring Shortcut Learning using Mutual Information27 Jun 2022
- 2022Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling15 Jun 2022
- 2022Machine Learning and Deep Learning -- A review for Ecologists11 Apr 2022
- 2021PAC-Bayes Information Bottleneck29 Sep 2021
- 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronization30 Aug 2021
12 newest of 14 All 14 (JSON)