# Adversarial Robustness of Deep Neural Networks: A Survey from a Formal Verification Perspective

arXiv: https://arxiv.org/abs/2206.12227 · HTML: https://mlsanity.com/paper/2206.12227/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 104 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 30 of 45)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Efficient Neural Network Robustness Certification with General Activation Functions](https://mlsanity.com/paper/1811.00866/) (2018)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [Towards Fast Computation of Certified Robustness for ReLU Networks](https://mlsanity.com/paper/1804.09699/) (2018)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks](https://mlsanity.com/paper/1705.01320/) (2017)
- [Algorithms for Verifying Deep Neural Networks](https://mlsanity.com/paper/1903.06758/) (2019)
- [Formal Security Analysis of Neural Networks using Symbolic Intervals](https://mlsanity.com/paper/1804.10829/) (2018)
- [Semidefinite relaxations for certifying robustness to adversarial examples](https://mlsanity.com/paper/1811.01057/) (2018)
- [Efficient Formal Safety Analysis of Neural Networks](https://mlsanity.com/paper/1809.08098/) (2018)
- [NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems](https://mlsanity.com/paper/2004.05519/) (2020)
- [An approach to reachability analysis for feed-forward ReLU neural networks](https://mlsanity.com/paper/1706.07351/) (2017)
- [A Dual Approach to Scalable Verification of Deep Networks](https://mlsanity.com/paper/1803.06567/) (2018)
- [Globally-Robust Neural Networks](https://mlsanity.com/paper/2102.08452/) (2021)
- [SoK: Certified Robustness for Deep Neural Networks](https://mlsanity.com/paper/2009.04131/) (2020)
- [Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming](https://mlsanity.com/paper/1903.01287/) (2019)
- [Maximum Resilience of Artificial Neural Networks](https://mlsanity.com/paper/1705.01040/) (2017)
- [Reachability Analysis of Deep Neural Networks with Provable Guarantees](https://mlsanity.com/paper/1805.02242/) (2018)
- [Measuring Neural Net Robustness with Constraints](https://mlsanity.com/paper/1605.07262/) (2016)

## Built on by (most cited first, 30 of 42)

- [Ctrl-Z: Controlling AI Agents via Resampling](https://mlsanity.com/paper/2504.10374/) (2025)
- [ToolGate: Contract-Grounded and Verified Tool Execution for LLMs](https://mlsanity.com/paper/2601.04688/) (2026)
- [From Robustness to Explainability and Back Again](https://mlsanity.com/paper/2306.03048/) (2023)
- [How Secure Are Large Language Models (LLMs) for Navigation in Urban Environments?](https://mlsanity.com/paper/2402.09546/) (2024)
- [Machine Learning Robustness: A Primer](https://mlsanity.com/paper/2404.00897/) (2024)
- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [On the Robustness of Explanations of Deep Neural Network Models: A Survey](https://mlsanity.com/paper/2211.04780/) (2022)
- [Towards Certification of Uncertainty Calibration under Adversarial Attacks](https://mlsanity.com/paper/2405.13922/) (2024)
- [Model Monitoring and Robustness of In-Use Machine Learning Models: Quantifying Data Distribution Shifts Using Population Stability Index](https://mlsanity.com/paper/2302.00775/) (2023)
- [Certified Robust Invariant Polytope Training in Neural Controlled ODEs](https://mlsanity.com/paper/2408.01273/) (2024)
- [Monitoring Robustness and Individual Fairness](https://mlsanity.com/paper/2506.00496/) (2025)
- [Robustness Verification of Polynomial Neural Networks](https://mlsanity.com/paper/2602.06105/) (2026)
- [Verification of Neural Networks against Convolutional Perturbations via Parameterised Kernels](https://mlsanity.com/paper/2411.04594/) (2024)
- [An Investigation of Visual Foundation Models Robustness](https://mlsanity.com/paper/2508.16225/) (2025)
- [Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06921/) (2025)
- [Tightening Robustness Verification of MaxPool-based Neural Networks via Minimizing the Over-Approximation Zone](https://mlsanity.com/paper/2211.09810/) (2022)
- [A Training Rate and Survival Heuristic for Inference and Robustness Evaluation (TRASHFIRE)](https://mlsanity.com/paper/2401.13751/) (2024)
- [Reasoning Under Threat: Symbolic and Neural Techniques for Cybersecurity Verification](https://mlsanity.com/paper/2503.22755/) (2025)
- [A Formally Verified Robustness Certifier for Neural Networks (Extended Version)](https://mlsanity.com/paper/2505.06958/) (2025)
- [Non-Parametric Probabilistic Robustness: A Conservative Risk Estimator under Unknown Perturbation Distributions](https://mlsanity.com/paper/2511.17380/) (2025)
- [On AI Safety and Security Technical Debt in Engineering AI-Enabled Systems](https://mlsanity.com/paper/2607.23365/) (2026)
- [Towards General Robustness Verification of MaxPool-based Convolutional Neural Networks via Tightening Linear Approximation](https://mlsanity.com/paper/2406.00699/) (2024)
- [Data-driven Verification of DNNs for Object Recognition](https://mlsanity.com/paper/2408.00783/) (2024)
- [A practical approach to evaluating the adversarial distance for machine learning classifiers](https://mlsanity.com/paper/2409.03598/) (2024)
- [Detecting underdetermination in parameterized quantum circuits](https://mlsanity.com/paper/2504.03315/) (2025)
- [SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments](https://mlsanity.com/paper/2504.21454/) (2025)
- [DP-TRAE: A Dual-Phase Merging Transferable Reversible Adversarial Example for Image Privacy Protection](https://mlsanity.com/paper/2505.06860/) (2025)
- [Probably Approximately Global Robustness Certification](https://mlsanity.com/paper/2511.06495/) (2025)
- [SMiLE: Provably Enforcing Global Relational Properties in Neural Networks](https://mlsanity.com/paper/2511.07208/) (2025)
- [The Cost of Relaxation: Evaluating the Error in Convex Neural Network Verification](https://mlsanity.com/paper/2604.18728/) (2026)

## Abstract (arXiv)

Neural networks have been widely applied in security applications such as spam and phishing detection, intrusion prevention, and malware detection. This black-box method, however, often has uncertainty and poor explainability in applications. Furthermore, neural networks themselves are often vulnerable to adversarial attacks. For those reasons, there is a high demand for trustworthy and rigorous methods to verify the robustness of neural network models. Adversarial robustness, which concerns the reliability of a neural network when dealing with maliciously manipulated inputs, is one of the hottest topics in security and machine learning. In this work, we survey existing literature in adversarial robustness verification for neural networks and collect 39 diversified research works across machine learning, security, and software engineering domains. We systematically analyze their approaches, including how robustness is formulated, what verification techniques are used, and the strengths and limitations of each technique. We provide a taxonomy from a formal verification perspective for a comprehensive understanding of this topic. We classify the existing techniques based on property specification, problem reduction, and reasoning strategies. We also demonstrate representative techniques that have been applied in existing studies with a sample model. Finally, we discuss open questions for future research.