# Trust but Verify: An Information-Theoretic Explanation for the Adversarial Fragility of Machine Learning Systems, and a General Defense against Adversarial Attacks

arXiv: https://arxiv.org/abs/1905.11381 · HTML: https://mlsanity.com/paper/1905.11381/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2019.

## Built on by (most cited first, 4 of 4)

- [An Adaptive Black-box Defense against Trojan Attacks (TrojDef)](https://mlsanity.com/paper/2209.01721/) (2022)
- [Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems](https://mlsanity.com/paper/2209.10058/) (2022)
- [Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems](https://mlsanity.com/paper/2210.01000/) (2022)
- [Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systems](https://mlsanity.com/paper/2211.12685/) (2022)

## Abstract (arXiv)

Deep-learning based classification algorithms have been shown to be susceptible to adversarial attacks: minor changes to the input of classifiers can dramatically change their outputs, while being imperceptible to humans. In this paper, we present a simple hypothesis about a feature compression property of artificial intelligence (AI) classifiers and present theoretical arguments to show that this hypothesis successfully accounts for the observed fragility of AI classifiers to small adversarial perturbations. Drawing on ideas from information and coding theory, we propose a general class of defenses for detecting classifier errors caused by abnormally small input perturbations. We further show theoretical guarantees for the performance of this detection method. We present experimental results with (a) a voice recognition system, and (b) a digit recognition system using the MNIST database, to demonstrate the effectiveness of the proposed defense methods. The ideas in this paper are motivated by a simple analogy between AI classifiers and the standard Shannon model of a communication system.