mlsanityData to 7 Oct 2026

Adversarial Information Bottleneck

arXiv 2103.00381 ↗First posted 28 Feb 2021cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

The information bottleneck (IB) principle has been adopted to explain deep learning in terms of information compression and prediction, which are balanced by a trade-off hyperparameter. How to optimize the IB principle for better robustness and figure out the effects of compression through the trade-off hyperparameter are two challenging problems. Previous methods attempted to optimize the IB principle by introducing random noise into learning the representation and achieved state-of-the-art performance in the nuisance information compression and semantic information extraction. However, their performance on resisting adversarial perturbations is far less impressive. To this end, we propose an adversarial information bottleneck (AIB) method without any explicit assumptions about the underlying distribution of the representations, which can be optimized effectively by solving a Min-Max optimization problem. Numerical experiments on synthetic and real-world datasets demonstrate its effectiveness on learning more invariant representations and mitigating adversarial perturbations compared to several competing IB methods. In addition, we analyse the adversarial robustness of diverse IB methods contrasting with their IB curves, and reveal that IB models with the hyperparameter β corresponding to the knee point in the IB curve achieve the best trade-off between compression and prediction, and has best robustness against various attacks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 28 Feb 2021.

Builds on

30 of 55 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  8. 2016Deep Variational Information Bottleneckcited by 915
  9. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  11. 2018MINE: Mutual Information Neural Estimationcited by 680
  12. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
18 more
  1. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  2. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  3. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  4. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  5. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  6. 2016Technical Report on the CleverHans v2.1.0 Adversarial Examples Librarycited by 123
  7. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  8. 2020The Conditional Entropy Bottleneckcited by 75
  9. 2017Nonlinear Information Bottleneckcited by 71
  10. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  11. 2017Dual Discriminator Generative Adversarial Netscited by 55
  12. 2019Learnability for the Information Bottleneckcited by 23
  13. 2016Relevant sparse codes with variational information bottleneckcited by 17
  14. 2017Optimal rates of entropy estimation over Lipschitz ballscited by 15
  15. 2019Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernelscited by 11
  16. 2019Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Codingcited by 8
  17. 2017Geometric k-nearest neighbor estimation of entropy and mutual informationcited by 7
  18. 2020An Information-Theoretic Perspective on Overfitting and Underfittingcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 3.

5 papers.

  1. 2024Progressive Feedforward Collapse of ResNet Trainingcited by 12
  2. 2024REMEDI: Corrective Transformations for Improved Neural Entropy Estimationcited by 2
  3. 2022Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneckcited by 1
  4. 2024OTAD: An Optimal Transport-Induced Robust Model for Agnostic Adversarial Attackcited by 1
  5. 2022Understanding Adversarial Robustness from Feature Maps of Convolutional Layerscited by 0