mlsanityData to 7 Oct 2026

Deep Learning and the Information Bottleneck Principle

arXiv 1503.02406 ↗First posted 9 Mar 2015cs.LGcited by 793

Not on the map. Why?

Abstract (arXiv)

Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.

Key facts. Cited by 793 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 1; 2021: 91; 2022: 114; 2023: 136; 2024: 145; 2025: 172; 2026: 133.

793 papers.

Sort:

By area:

  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  4. 2022A Survey of Machine Unlearningcited by 298
  5. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  6. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  7. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  8. 2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  10. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  11. 2022Understanding The Robustness in Vision Transformerscited by 144
  12. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
38 more
  1. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  2. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  3. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  4. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  5. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  6. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  7. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  8. 2024A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networkscited by 103
  9. 2022Out-Of-Distribution Generalization on Graphs: A Surveycited by 97
  10. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  11. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  12. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  13. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  14. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  15. 2021SE(3) Equivariant Graph Neural Networks with Complete Local Framescited by 75
  16. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  17. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  18. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 67
  19. 2021Reducing Information Bottleneck for Weakly Supervised Semantic Segmentationcited by 65
  20. 2021Understanding Softmax Confidence and Uncertaintycited by 62
  21. 2022Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Offcited by 61
  22. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  23. 2024Parallelized Autoregressive Visual Generationcited by 58
  24. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  25. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  26. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  27. 2021Negative Data Augmentationcited by 46
  28. 2022A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Predictioncited by 46
  29. 2021Frustratingly Easy Transferability Estimationcited by 44
  30. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  31. 2021Compressive Visual Representationscited by 41
  32. 2023Towards Self-Interpretable Graph-Level Anomaly Detectioncited by 41
  33. 2024DRCT: Saving Image Super-resolution away from Information Bottleneckcited by 40
  34. 2022ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networkscited by 39
  35. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  36. 2024IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistencycited by 38
  37. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  38. 2020Intrinsic Reward Driven Imitation Learning via Generative Modelcited by 35

50 most cited of 793 All 793 (JSON)