mlsanityData to 7 Oct 2026

Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional

arXiv 2102.00533 ↗First posted 31 Jan 2021cs.LG cs.IT math.ITcited by 18

Not on the map. Why?

Abstract (arXiv)

We introduce the matrix-based Renyi's α-order entropy functional to parameterize Tishby et al. information bottleneck (IB) principle with a neural network. We term our methodology Deep Deterministic Information Bottleneck (DIB), as it avoids variational inference and distribution assumption. We show that deep neural networks trained with DIB outperform the variational objective counterpart and those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.Code available at https://github.com/yuxi120407/DIB

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 31 Jan 2021.

Builds on

20 of 34 references.

References · 20

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2016Deep Variational Information Bottleneckcited by 915
  7. 2018MINE: Mutual Information Neural Estimationcited by 680
  8. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  9. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  10. 2018Understanding disentangling in β-VAEcited by 332
  11. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  12. 2017Nonlinear Information Bottleneckcited by 71
8 more
  1. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  2. 2020The Information Bottleneck Problem and Its Applications in Machine Learningcited by 66
  3. 2016The deterministic information bottleneckcited by 42
  4. 2018Caveats for information bottleneck in deterministic scenarioscited by 34
  5. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 31
  6. 2020On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Viewscited by 28
  7. 2018Multivariate Extension of Matrix-based Renyi's α-order Entropy Functionalcited by 5
  8. 2020Interpretable Fault Detection using Projections of Mutual Information Matrixcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 7; 2023: 5; 2024: 1; 2025: 2.

18 papers.

Sort:

By area:

  1. 2021Information Theoretic Representation Distillationcited by 19
  2. 2023MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustnesscited by 12
  3. 2021Understanding Neural Networks with Logarithm Determinant Entropy Estimatorcited by 10
  4. 2022R2-Trans:Fine-Grained Visual Categorization with Redundancy Reductioncited by 8
  5. 2022Optimal Randomized Approximations for Matrix based Renyi's Entropycited by 7
  6. 2022Multi-view Information Bottleneck Without Variational Approximationcited by 5
  7. 2023Recognizable Information Bottleneckcited by 5
  8. 2021Computationally Efficient Approximations for Matrix-based Renyi's Entropycited by 4
  9. 2023Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performancecited by 3
  10. 2023Filter Pruning based on Information Capacity and Independencecited by 2
  11. 2023Local Intrinsic Dimensional Entropycited by 2
  12. 2022Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneckcited by 1
6 more
  1. 2022Deep Supervised Information Bottleneck Hashing for Cross-modal Retrieval based Computer-aided Diagnosiscited by 1
  2. 2022Robust and Fast Measure of Information via Low-rank Representationcited by 1
  3. 2024Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Trainingcited by 1
  4. 2022Mutual information estimation for graph convolutional neural networkscited by 0
  5. 2025Label Smoothing is a Pragmatic Information Bottleneckcited by 0
  6. 2025Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognitioncited by 0