# Adversarial Information Bottleneck

arXiv: https://arxiv.org/abs/2103.00381 · HTML: https://mlsanity.com/paper/2103.00381/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 28 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Towards Deep Neural Network Architectures Robust to Adversarial Examples](https://mlsanity.com/paper/1412.5068/) (2014)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack](https://mlsanity.com/paper/1811.09310/) (2018)
- [Technical Report on the CleverHans v2.1.0 Adversarial Examples Library](https://mlsanity.com/paper/1610.00768/) (2016)
- [The Conditional Entropy Bottleneck](https://mlsanity.com/paper/2002.05379/) (2020)
- [Nonlinear Information Bottleneck](https://mlsanity.com/paper/1705.02436/) (2017)
- [Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle](https://mlsanity.com/paper/1802.09766/) (2018)
- [Dual Discriminator Generative Adversarial Nets](https://mlsanity.com/paper/1709.03831/) (2017)
- [Learnability for the Information Bottleneck](https://mlsanity.com/paper/1907.07331/) (2019)
- [Relevant sparse codes with variational information bottleneck](https://mlsanity.com/paper/1605.07332/) (2016)
- [Optimal rates of entropy estimation over Lipschitz balls](https://mlsanity.com/paper/1711.02141/) (2017)
- [Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels](https://mlsanity.com/paper/1909.11396/) (2019)
- [Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Coding](https://mlsanity.com/paper/1912.02605/) (2019)
- [Geometric k-nearest neighbor estimation of entropy and mutual information](https://mlsanity.com/paper/1711.00748/) (2017)
- [An Information-Theoretic Perspective on Overfitting and Underfitting](https://mlsanity.com/paper/2010.06076/) (2020)

## Built on by (most cited first, 5 of 5)

- [Progressive Feedforward Collapse of ResNet Training](https://mlsanity.com/paper/2405.00985/) (2024)
- [REMEDI: Corrective Transformations for Improved Neural Entropy Estimation](https://mlsanity.com/paper/2402.05718/) (2024)
- [Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck](https://mlsanity.com/paper/2203.02592/) (2022)
- [OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attack](https://mlsanity.com/paper/2408.00329/) (2024)
- [Understanding Adversarial Robustness from Feature Maps of Convolutional Layers](https://mlsanity.com/paper/2202.12435/) (2022)

## Abstract (arXiv)

The information bottleneck (IB) principle has been adopted to explain deep learning in terms of information compression and prediction, which are balanced by a trade-off hyperparameter. How to optimize the IB principle for better robustness and figure out the effects of compression through the trade-off hyperparameter are two challenging problems. Previous methods attempted to optimize the IB principle by introducing random noise into learning the representation and achieved state-of-the-art performance in the nuisance information compression and semantic information extraction. However, their performance on resisting adversarial perturbations is far less impressive. To this end, we propose an adversarial information bottleneck (AIB) method without any explicit assumptions about the underlying distribution of the representations, which can be optimized effectively by solving a Min-Max optimization problem. Numerical experiments on synthetic and real-world datasets demonstrate its effectiveness on learning more invariant representations and mitigating adversarial perturbations compared to several competing IB methods. In addition, we analyse the adversarial robustness of diverse IB methods contrasting with their IB curves, and reveal that IB models with the hyperparameter β corresponding to the knee point in the IB curve achieve the best trade-off between compression and prediction, and has best robustness against various attacks.