# DeepSafe: A Data-driven Approach for Checking Adversarial Robustness in Neural Networks

arXiv: https://arxiv.org/abs/1710.00486 · HTML: https://mlsanity.com/paper/1710.00486/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2017.

## Built on by (most cited first, 20 of 20)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)](https://mlsanity.com/paper/2102.01564/) (2021)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Formally Explaining Neural Networks within Reactive Systems](https://mlsanity.com/paper/2308.00143/) (2023)
- [Verification-Aided Deep Ensemble Selection](https://mlsanity.com/paper/2202.03898/) (2022)
- [Closed-loop Analysis of Vision-based Autonomous Systems: A Case Study](https://mlsanity.com/paper/2302.04634/) (2023)
- [Minimal Multi-Layer Modifications of Deep Neural Networks](https://mlsanity.com/paper/2110.09929/) (2021)
- [Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance](https://mlsanity.com/paper/2304.06715/) (2023)
- [Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice](https://mlsanity.com/paper/2102.01979/) (2021)
- [DelBugV: Delta-Debugging Neural Network Verifiers](https://mlsanity.com/paper/2305.18558/) (2023)
- [A Survey of Neural Network Robustness Assessment in Image Recognition](https://mlsanity.com/paper/2404.08285/) (2024)
- [Assumption Generation for the Verification of Learning-Enabled Autonomous Systems](https://mlsanity.com/paper/2305.18372/) (2023)
- [ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs](https://mlsanity.com/paper/2410.24214/) (2024)
- [Scalable Whitebox Attacks on Tree-based Models](https://mlsanity.com/paper/2204.00103/) (2022)
- [On the Relationship Between RNN Hidden State Vectors and Semantic Ground Truth](https://mlsanity.com/paper/2306.16854/) (2023)
- [Scaling Model Checking for DNN Analysis via State-Space Reduction and Input Segmentation (Extended Version)](https://mlsanity.com/paper/2306.17323/) (2023)
- [DeepKnowledge: Generalisation-Driven Deep Learning Testing](https://mlsanity.com/paper/2403.16768/) (2024)
- [Clustered Calibration: Representation-Aware Probability Calibration via Learned Subpopulations](https://mlsanity.com/paper/2510.19328/) (2025)
- [Enhancing Adversarial Robustness of IoT Intrusion Detection via SHAP-Based Attribution Fingerprinting](https://mlsanity.com/paper/2511.06197/) (2025)

## Abstract (arXiv)

Deep neural networks have become widely used, obtaining remarkable results in domains such as computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, and bio-informatics, where they have produced results comparable to human experts. However, these networks can be easily fooled by adversarial perturbations: minimal changes to correctly-classified inputs, that cause the network to mis-classify them. This phenomenon represents a concern for both safety and security, but it is currently unclear how to measure a network's robustness against such perturbations. Existing techniques are limited to checking robustness around a few individual input points, providing only very limited guarantees. We propose a novel approach for automatically identifying safe regions of the input space, within which the network is robust against adversarial perturbations. The approach is data-guided, relying on clustering to identify well-defined geometric regions as candidate safe regions. We then utilize verification techniques to confirm that these regions are safe or to provide counter-examples showing that they are not safe. We also introduce the notion of targeted robustness which, for a given target label and region, ensures that a NN does not map any input in the region to the target label. We evaluated our technique on the MNIST dataset and on a neural network implementation of a controller for the next-generation Airborne Collision Avoidance System for unmanned aircraft (ACAS Xu). For these networks, our approach identified multiple regions which were completely safe as well as some which were only safe for specific labels. It also discovered several adversarial perturbations of interest.