mlsanityData to 14 Oct 2026

How Does Information Bottleneck Help Deep Learning?

arXiv 2305.18887 ↗First posted 30 May 2023cs.LG cs.AI cs.CLcited by 93

Not on the map. Why?

Abstract (arXiv)

Numerous deep learning algorithms have been inspired by and understood via the notion of information bottleneck, where unnecessary information is (often implicitly) minimized while task-relevant information is maximized. However, a rigorous argument for justifying why it is desirable to control information bottlenecks has been elusive. In this paper, we provide the first rigorous learning theory for justifying the benefit of information bottleneck in deep learning by mathematically relating information bottleneck to generalization errors. Our theory proves that controlling information bottleneck is one way to control generalization errors in deep learning, although it is not the only or necessary way. We investigate the merit of our new mathematical findings with experiments across a range of architectures and learning settings. In many cases, generalization errors are shown to correlate with the degree of information bottleneck: i.e., the amount of the unnecessary information at hidden layers. This paper provides a theoretical foundation for current and future methods through the lens of information bottleneck. Our new generalization bounds scale with the degree of information bottleneck, unlike the previous bounds that scale with the number of parameters, VC dimension, Rademacher complexity, stability or robustness. Our code is publicly available at: https://github.com/xu-ji/information-bottleneck

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

48 of 68 references.

References · 48

  1. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  2. 2016Deep Variational Information Bottleneckcited by 930
  3. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  5. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  6. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  7. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  8. 2017Mitigating Adversarial Effects Through Randomizationcited by 383
  9. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  10. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  11. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  12. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 293
36 more
  1. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  2. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  3. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
  4. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  5. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 191
  6. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  7. 2017Generalization in Deep Learningcited by 146
  8. 2020Training robust neural networks using Lipschitz boundscited by 99
  9. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  10. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  11. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  12. 2021Graph Structure Learning with Variational Information Bottleneckcited by 81
  13. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  14. 2020The Conditional Entropy Bottleneckcited by 75
  15. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  16. 2020On the role of data in PAC-Bayes boundscited by 64
  17. 2019Robustness Certificates for Sparse Adversarial Attacks by Randomized Ablationcited by 60
  18. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 58
  19. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  20. 2020Generalization Bounds via Information Density and Conditional Information Densitycited by 43
  21. 2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 42
  22. 2021Compressive Visual Representationscited by 41
  23. 2021Discrete-Valued Neural Communicationcited by 39
  24. 2020Randomization matters. How to defend against strong adversarial attackscited by 37
  25. 2017Estimating Mixture Entropy with Pairwise Distancescited by 36
  26. 2019Theoretical evidence for adversarial robustness through randomizationcited by 33
  27. 2021Robust Predictable Controlcited by 29
  28. 2020Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constantcited by 25
  29. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  30. 2022Discrete Key-Value Bottleneckcited by 18
  31. 2022On Rademacher Complexity-based Generalization Bounds for Deep Learningcited by 16
  32. 2019Adaptive Estimators Show Information Compression in Deep Neural Networkscited by 14
  33. 2022Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantizationcited by 11
  34. 2021Scaffolding Setscited by 8
  35. 2022Bounding generalization error with input compression: An empirical study with infinite-width networkscited by 4
  36. 2022Explanation Regeneration via Information Bottleneckcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 25; 2025: 39; 2026: 22.

93 papers.

Sort:

By area:

  1. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  2. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  3. 2024Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoderscited by 38
  4. 2024The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipelinecited by 33
  5. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  6. 2023Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networkscited by 23
  7. 2023On Copyright Risks of Text-to-Image Diffusion Modelscited by 20
  8. 2023Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEscited by 19
  9. 2024Can AI Be as Creative as Humans?cited by 17
  10. 2024An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generationcited by 16
  11. 2024Minimum Description Length and Generalization Guarantees for Representation Learningcited by 11
  12. 2025On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passingcited by 11
38 more
  1. 2024Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embeddingcited by 10
  2. 2025Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulationcited by 10
  3. 2025Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformerscited by 10
  4. 2024Cauchy-Schwarz Divergence Information Bottleneck for Regressioncited by 9
  5. 2024End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Trainingcited by 8
  6. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  7. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  8. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 6
  9. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 6
  10. 2023VA3: Virtually Assured Amplification Attack on Probabilistic Copyright Protection for Text-to-Image Generative Modelscited by 5
  11. 2024Unified Source-Free Domain Adaptationcited by 5
  12. 2024Pretraining with Random Noise for Fast and Robust Learning without Weight Transportcited by 5
  13. 2025Visual Instruction Bottleneck Tuningcited by 5
  14. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
  15. 2024Deep Regression Representation Learning with Topologycited by 4
  16. 2024PAC-Bayes Analysis for Recalibration in Classificationcited by 4
  17. 2024Generalization Bounds via Conditional f-Informationcited by 4
  18. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  19. 2025A Generalized Information Bottleneck Theory of Deep Learningcited by 4
  20. 2026CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learningcited by 4
  21. 2023Learning Unorthogonalized Matrices for Rotation Estimationcited by 3
  22. 2024Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworkscited by 3
  23. 2025AIF-SFDA: Autonomous Information Filter-driven Source-Free Domain Adaptation for Medical Image Segmentationcited by 3
  24. 2025Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoningcited by 3
  25. 2025VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DRcited by 3
  26. 2025Prepare Before You Act: Learning From Humans to Rearrange Initial Statescited by 3
  27. 2025Generalization or Memorization: Dynamic Decoding for Mode Steeringcited by 3
  28. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  29. 2024Class-wise Generalization Error: an Information-Theoretic Analysiscited by 2
  30. 2024Representation Learning with Conditional Information Flow Maximizationcited by 2
  31. 2024Steinmetz Neural Networks for Complex-Valued Datacited by 2
  32. 2024Entropy, concentration, and learning: a statistical mechanics primercited by 2
  33. 2025Improving Deep Regression with Tightnesscited by 2
  34. 2025Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorscited by 2
  35. 2025Generalization Capability for Imitation Learningcited by 2
  36. 2025JigsawComm: Joint Semantic Feature Encoding and Transmission for Communication-Efficient Cooperative Perceptioncited by 2
  37. 2025Explainable Multimodal Regression via Information Decompositioncited by 2
  38. 2026Tackling the Inherent Difficulty of Noise Filtering in RAGcited by 2

50 most cited of 93 All 93 (JSON)