mlsanityData to 14 Oct 2026

A Generalized Information Bottleneck Theory of Deep Learning

arXiv 2509.26327 ↗First posted 30 Sep 2025cs.LG cs.IT math.ITcited by 4

Not on the map. Why?

Abstract (arXiv)

The Information Bottleneck (IB) principle offers a compelling theoretical framework to understand how neural networks (NNs) learn. However, its practical utility has been constrained by unresolved theoretical ambiguities and significant challenges in accurate estimation. In this paper, we present a textitGeneralized Information Bottleneck (GIB) framework that reformulates the original IB principle through the lens of synergy, i.e., the information obtainable only through joint processing of features. We provide theoretical and empirical evidence demonstrating that synergistic functions achieve superior generalization compared to their non-synergistic counterparts. Building on these foundations we re-formulate the IB using a computable definition of synergy based on the average interaction information (II) of each feature with those remaining. We demonstrate that the original IB objective is upper bounded by our GIB in the case of perfect estimation, ensuring compatibility with existing IB theory while addressing its limitations. Our experimental results demonstrate that GIB consistently exhibits compression phases across a wide range of architectures (including those with textitReLU activations where the standard IB fails), while yielding interpretable dynamics in both CNNs and Transformers and aligning more closely with our understanding of adversarial robustness.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 17 of its 37 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

17 of 37 references.

References · 17

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2016Deep Variational Information Bottleneckcited by 930
  3. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  4. 2018MINE: Mutual Information Neural Estimationcited by 683
  5. 2017Exploring Generalization in Deep Learningcited by 600
  6. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  7. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  8. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  9. 2017Nonlinear Information Bottleneckcited by 71
  10. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 37
  11. 2019Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Informationcited by 30
  12. 2021Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustnesscited by 20
5 more
  1. 2013Multivariate Information Bottleneckcited by 17
  2. 2018BROJA-2PID: A robust estimator for bivariate partial information decompositioncited by 6
  3. 2024Partial Information Decomposition for Data Interpretability and Feature Selectioncited by 4
  4. 2024Partial information decomposition: redundancy as information bottleneckcited by 3
  5. 2019MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decompositioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  2. 2025Generalization or Memorization: Dynamic Decoding for Mode Steeringcited by 3
  3. 2026Implicit Hypothesis Testing and Divergence Preservation in Neural Network Representationscited by 0
  4. 2026The Sufficiency-Conciseness Trade-off in LLM Self-Explanation from an Information Bottleneck Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression
  2. 2021Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
  3. 2024Unlearning Information Bottleneck: Machine Unlearning of Systematic Patterns and Biases
  4. 2025Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models
  5. 2026There Will Be a Scientific Theory of Deep Learning
  6. 2021Information Bottleneck Approach to Spatial Attention Learning
  7. 2023Information-Ordered Bottlenecks for Adaptive Semantic Compression
  8. 2023On information captured by neural networks: connections with memorization and generalization