# Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples

arXiv: https://arxiv.org/abs/1708.05493 · HTML: https://mlsanity.com/paper/1708.05493/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2017.

## Built on by (most cited first, 27 of 27)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Rethinking Model Ensemble in Transfer-based Adversarial Attacks](https://mlsanity.com/paper/2303.09105/) (2023)
- [Zero-shot Adversarial Quantization](https://mlsanity.com/paper/2103.15263/) (2021)
- [Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks](https://mlsanity.com/paper/2104.05097/) (2021)
- [Robust Feature-Level Adversaries are Interpretability Tools](https://mlsanity.com/paper/2110.03605/) (2021)
- [Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey](https://mlsanity.com/paper/2410.23687/) (2024)
- [A Unified Game-Theoretic Interpretation of Adversarial Robustness](https://mlsanity.com/paper/2111.03536/) (2021)
- [IG2: Integrated Gradient on Iterative Gradient Path for Feature Attribution](https://mlsanity.com/paper/2406.10852/) (2024)
- [DI-AA: An Interpretable White-box Attack for Fooling Deep Neural Networks](https://mlsanity.com/paper/2110.07305/) (2021)
- [Scaling Laws for Black box Adversarial Attacks](https://mlsanity.com/paper/2411.16782/) (2024)
- [Interpretable Machine Learning for Discovery: Statistical Challenges & Opportunities](https://mlsanity.com/paper/2308.01475/) (2023)
- [Evaluating the Robustness of Bayesian Neural Networks Against Different Types of Attacks](https://mlsanity.com/paper/2106.09223/) (2021)
- [Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks](https://mlsanity.com/paper/2211.10024/) (2022)
- [EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry](https://mlsanity.com/paper/2103.08820/) (2021)
- [It is not "accuracy vs. explainability" -- we need both for trustworthy AI systems](https://mlsanity.com/paper/2212.11136/) (2022)
- [An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotations](https://mlsanity.com/paper/2505.16705/) (2025)
- [Learning Minimal Neural Specifications](https://mlsanity.com/paper/2404.04662/) (2024)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [Resilience of Bayesian Layer-Wise Explanations under Adversarial Attacks](https://mlsanity.com/paper/2102.11010/) (2021)
- [A Unified Game-Theoretic Interpretation of Adversarial Robustness](https://mlsanity.com/paper/2103.07364/) (2021)
- [Noise Modulation: Let Your Model Interpret Itself](https://mlsanity.com/paper/2103.10603/) (2021)
- [Prediction of Hereditary Cancers Using Neural Networks](https://mlsanity.com/paper/2106.13682/) (2021)
- [Multi-concept adversarial attacks](https://mlsanity.com/paper/2110.10287/) (2021)
- [Feature Chirality in Deep Learning Models](https://mlsanity.com/paper/2305.03966/) (2023)
- [Adversarial Doodles: Interpretable and Human-drawable Attacks Provide Describable Insights](https://mlsanity.com/paper/2311.15994/) (2023)
- [Robust Explainable Recommendation](https://mlsanity.com/paper/2405.01855/) (2024)

## Abstract (arXiv)

Deep neural networks (DNNs) have demonstrated impressive performance on a wide array of tasks, but they are usually considered opaque since internal structure and learned parameters are not interpretable. In this paper, we re-examine the internal representations of DNNs using adversarial images, which are generated by an ensemble-optimization algorithm. We find that: (1) the neurons in DNNs do not truly detect semantic objects/parts, but respond to objects/parts only as recurrent discriminative patches; (2) deep visual representations are not robust distributed codes of visual concepts because the representations of adversarial images are largely not consistent with those of real images, although they have similar visual appearance, both of which are different from previous findings. To further improve the interpretability of DNNs, we propose an adversarial training scheme with a consistent loss such that the neurons are endowed with human-interpretable concepts. The induced interpretable representations enable us to trace eventual outcomes back to influential neurons. Therefore, human users can know how the models make predictions, as well as when and why they make errors.