# Benchmarking Robustness of Deep Learning Classifiers Using Two-Factor Perturbation

arXiv: https://arxiv.org/abs/2103.03102 · HTML: https://mlsanity.com/paper/2103.03102/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 15 of 15)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [Towards Stable and Efficient Training of Verifiably Robust Neural Networks](https://mlsanity.com/paper/1906.06316/) (2019)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Understanding How Image Quality Affects Deep Neural Networks](https://mlsanity.com/paper/1604.04004/) (2016)
- [A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions](https://mlsanity.com/paper/1705.02498/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations](https://mlsanity.com/paper/1807.01697/) (2018)
- [Examining the Impact of Blur on Recognition by Convolutional Networks](https://mlsanity.com/paper/1611.05760/) (2016)
- [Google's Cloud Vision API Is Not Robust To Noise](https://mlsanity.com/paper/1704.05051/) (2017)
- [Comparing deep neural networks against humans: object recognition when the signal gets weaker](https://mlsanity.com/paper/1706.06969/) (2017)
- [Quality Resilient Deep Neural Networks](https://mlsanity.com/paper/1703.08119/) (2017)
- [Improving Data Quality through Deep Learning and Statistical Models](https://mlsanity.com/paper/1810.07132/) (2018)

## Built on by (most cited first, 3 of 3)

- [Investigating the Corruption Robustness of Image Classifiers with Random Lp-norm Corruptions](https://mlsanity.com/paper/2305.05400/) (2023)
- [Discovering Limitations of Image Quality Assessments with Noised Deep Learning Image Sets](https://mlsanity.com/paper/2210.10249/) (2022)
- [ASI: Accuracy-Stability Index for Evaluating Deep Learning Models](https://mlsanity.com/paper/2311.15332/) (2023)

## Abstract (arXiv)

This paper adds to the fundamental body of work on benchmarking the robustness of deep learning (DL) classifiers. We innovate a new benchmarking methodology to evaluate robustness of DL classifiers. Also, we introduce a new four-quadrant statistical visualization tool, including minimum accuracy, maximum accuracy, mean accuracy, and coefficient of variation, for benchmarking robustness of DL classifiers. To measure robust DL classifiers, we created a comprehensive 69 benchmarking image set, including a clean set, sets with single factor perturbations, and sets with two-factor perturbation conditions. After collecting experimental results, we first report that using two-factor perturbed images improves both robustness and accuracy of DL classifiers. The two-factor perturbation includes (1) two digital perturbations (salt & pepper noise and Gaussian noise) applied in both sequences, and (2) one digital perturbation (salt & pepper noise) and a geometric perturbation (rotation) applied in both sequences. All source codes, related image sets, and preliminary data, figures are shared on a GitHub website to support future academic research and industry projects. The web resources locate at https://github.com/caperock/robustai