mlsanityData to 14 Oct 2026

Tighter Bounds on the Information Bottleneck with Application to Deep Learning

arXiv 2402.07639 ↗First posted 12 Feb 2024cs.LG cs.AI cs.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep Neural Nets (DNNs) learn latent representations induced by their downstream task, objective function, and other parameters. The quality of the learned representations impacts the DNN's generalization ability and the coherence of the emerging latent space. The Information Bottleneck (IB) provides a hypothetically optimal framework for data modeling, yet it is often intractable. Recent efforts combined DNNs with the IB by applying VAE-inspired variational methods to approximate bounds on mutual information, resulting in improved robustness to adversarial attacks. This work introduces a new and tighter variational bound for the IB, improving performance of previous IB-inspired DNNs. These advancements strengthen the case for the IB and its variational approximations as a data modeling framework, and provide a simple method to significantly enhance the adversarial robustness of classifier DNNs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

15 of 28 references.

References · 15

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2013Auto-Encoding Variational Bayescited by 4,053
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2019An Introduction to Variational Autoencoderscited by 1,000
  7. 2016Deep Variational Information Bottleneckcited by 930
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  10. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  11. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  12. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
3 more
  1. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  2. 2017Gaussian Lower Bound for the Information Bottleneck Limitcited by 5
  3. 2019On the Difference Between the Information Bottleneck and the Deep Information Bottleneckcited by 5

Built on by

Not cited yet.