# Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations

arXiv: https://arxiv.org/abs/1807.01697 · HTML: https://mlsanity.com/paper/1807.01697/

## Key facts

- Cited by 99 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2018.

## Built on by (most cited first, 30 of 99)

- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning](https://mlsanity.com/paper/2506.06659/) (2025)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Distilling Model Failures as Directions in Latent Space](https://mlsanity.com/paper/2206.14754/) (2022)
- [Editing a classifier by rewriting its prediction rules](https://mlsanity.com/paper/2112.01008/) (2021)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023)
- [Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection](https://mlsanity.com/paper/2306.09158/) (2023)
- [Negative Data Augmentation](https://mlsanity.com/paper/2102.05113/) (2021)
- [Generalized Parametric Contrastive Learning](https://mlsanity.com/paper/2209.12400/) (2022)
- [In Search of Lost Online Test-time Adaptation: A Survey](https://mlsanity.com/paper/2310.20199/) (2023)
- [SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies](https://mlsanity.com/paper/2106.09678/) (2021)
- [Test-Time Robust Personalization for Federated Learning](https://mlsanity.com/paper/2205.10920/) (2022)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)
- [Can CNNs Be More Robust Than Transformers?](https://mlsanity.com/paper/2206.03452/) (2022)
- [Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation](https://mlsanity.com/paper/2302.07865/) (2023)
- [Dual Focal Loss for Calibration](https://mlsanity.com/paper/2305.13665/) (2023)
- [3DB: A Framework for Debugging Computer Vision Models](https://mlsanity.com/paper/2106.03805/) (2021)
- [COME: Test-time adaption by Conservatively Minimizing Entropy](https://mlsanity.com/paper/2410.10894/) (2024)
- [B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions](https://mlsanity.com/paper/2403.09346/) (2024)
- [Are Deep Neural Networks SMARTer than Second Graders?](https://mlsanity.com/paper/2212.09993/) (2022)
- [Decomposing and Editing Predictions by Modeling Model Computation](https://mlsanity.com/paper/2404.11534/) (2024)
- [FLAC: Fairness-Aware Representation Learning by Suppressing Attribute-Class Associations](https://mlsanity.com/paper/2304.14252/) (2023)
- [PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts](https://mlsanity.com/paper/2308.01313/) (2023)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [The Five-Dollar Model: Generating Game Maps and Sprites from Sentence Embeddings](https://mlsanity.com/paper/2308.04052/) (2023)
- [FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection](https://mlsanity.com/paper/2410.11397/) (2024)
- [Out-of-Distribution Detection Methods Answer the Wrong Questions](https://mlsanity.com/paper/2507.01831/) (2025)
- [Componential Prompt-Knowledge Alignment for Domain Incremental Learning](https://mlsanity.com/paper/2505.04575/) (2025)
- [Metaethical Perspectives on 'Benchmarking' AI Ethics](https://mlsanity.com/paper/2204.05151/) (2022)

## Abstract (arXiv)

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Unlike recent robustness research, this benchmark evaluates performance on commonplace corruptions not worst-case adversarial corruptions. We find that there are negligible changes in relative corruption robustness from AlexNet to ResNet classifiers, and we discover ways to enhance corruption robustness. Then we propose a new dataset called Icons-50 which opens research on a new kind of robustness, surface variation robustness. With this dataset we evaluate the frailty of classifiers on new styles of known objects and unexpected instances of known classes. We also demonstrate two methods that improve surface variation robustness. Together our benchmarks may aid future work toward networks that learn fundamental class structure and also robustly generalize.