# Analyzing the Noise Robustness of Deep Neural Networks

arXiv: https://arxiv.org/abs/2001.09395 · HTML: https://mlsanity.com/paper/2001.09395/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2020.

## Built on by (most cited first, 30 of 31)

- [A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking](https://mlsanity.com/paper/2302.14301/) (2023)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks](https://mlsanity.com/paper/2108.12931/) (2021)
- [Learning-From-Disagreement: A Model Comparison and Visual Analytics Framework](https://mlsanity.com/paper/2201.07849/) (2022)
- [Generalizing Adversarial Explanations with Grad-CAM](https://mlsanity.com/paper/2204.05427/) (2022)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models](https://mlsanity.com/paper/2404.08793/) (2024)
- [Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles](https://mlsanity.com/paper/2103.10229/) (2021)
- [A Visual Analytics System for Improving Attention-based Traffic Forecasting Models](https://mlsanity.com/paper/2208.04350/) (2022)
- [Interactive Analysis of CNN Robustness](https://mlsanity.com/paper/2110.07667/) (2021)
- [Visual Analytics of Neuron Vulnerability to Adversarial Attacks on Convolutional Neural Networks](https://mlsanity.com/paper/2303.02814/) (2023)
- [Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning](https://mlsanity.com/paper/2311.01016/) (2023)
- [SIM2REALVIZ: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation](https://mlsanity.com/paper/2109.11801/) (2021)
- [Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research](https://mlsanity.com/paper/2308.06290/) (2023)
- [Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation](https://mlsanity.com/paper/2506.22776/) (2025)
- [Testing Feedforward Neural Networks Training Programs](https://mlsanity.com/paper/2204.00694/) (2022)
- [A Practical Framework for Unsupervised Structure Preservation Medical Image Enhancement](https://mlsanity.com/paper/2304.01864/) (2023)
- [Panda or not Panda? Understanding Adversarial Attacks with Interactive Visualization](https://mlsanity.com/paper/2311.13656/) (2023)
- [Adversarial Attacks on Machine Learning-Aided Visualizations](https://mlsanity.com/paper/2409.02485/) (2024)
- ["Understanding Robustness Lottery": A Geometric Visual Comparative Analysis of Neural Network Pruning Approaches](https://mlsanity.com/paper/2206.07918/) (2022)
- [MiMICRI: Towards Domain-centered Counterfactual Explanations of Cardiovascular Image Classification Models](https://mlsanity.com/paper/2404.16174/) (2024)
- [ChannelExplorer: Exploring Class Separability Through Activation Channel Visualization](https://mlsanity.com/paper/2505.04647/) (2025)
- [NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models](https://mlsanity.com/paper/2509.03985/) (2025)
- [VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models](https://mlsanity.com/paper/2509.14571/) (2025)
- [CACTUS: Detecting and Resolving Conflicts in Objective Functions](https://mlsanity.com/paper/2103.07805/) (2021)
- [Bayesian Neural Networks: A Min-Max Game Framework](https://mlsanity.com/paper/2311.11126/) (2023)
- [Understanding the Role of Pathways in a Deep Neural Network](https://mlsanity.com/paper/2402.18132/) (2024)

## Abstract (arXiv)

Adversarial examples, generated by adding small but intentionally imperceptible perturbations to normal examples, can mislead deep neural networks (DNNs) to make incorrect predictions. Although much work has been done on both adversarial attack and defense, a fine-grained understanding of adversarial examples is still lacking. To address this issue, we present a visual analysis method to explain why adversarial examples are misclassified. The key is to compare and analyze the datapaths of both the adversarial and normal examples. A datapath is a group of critical neurons along with their connections. We formulate the datapath extraction as a subset selection problem and solve it by constructing and training a neural network. A multi-level visualization consisting of a network-level visualization of data flows, a layer-level visualization of feature maps, and a neuron-level visualization of learned features, has been designed to help investigate how datapaths of adversarial and normal examples diverge and merge in the prediction process. A quantitative evaluation and a case study were conducted to demonstrate the promise of our method to explain the misclassification of adversarial examples.