# Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts

arXiv: https://arxiv.org/abs/2107.12052 · HTML: https://mlsanity.com/paper/2107.12052/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 30 of 32)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [A Fourier Perspective on Model Robustness in Computer Vision](https://mlsanity.com/paper/1906.08988/) (2019)
- [Adversarial Examples Improve Image Recognition](https://mlsanity.com/paper/1911.09665/) (2019)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Billion-scale semi-supervised learning for image classification](https://mlsanity.com/paper/1905.00546/) (2019)
- [Benchmarking the Robustness of Semantic Segmentation Models](https://mlsanity.com/paper/1908.05005/) (2019)
- [On Robustness and Transferability of Convolutional Neural Networks](https://mlsanity.com/paper/2007.08558/) (2020)
- [MoPro: Webly Supervised Learning with Momentum Prototypes](https://mlsanity.com/paper/2009.07995/) (2020)
- [CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition](https://mlsanity.com/paper/1712.02463/) (2017)
- [Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?](https://mlsanity.com/paper/1909.02436/) (2019)
- [How is Gaze Influenced by Image Transformations? Dataset and Model](https://mlsanity.com/paper/1905.06803/) (2019)
- [Identifying Statistical Bias in Dataset Replication](https://mlsanity.com/paper/2005.09619/) (2020)
- [An Effective Anti-Aliasing Approach for Residual Networks](https://mlsanity.com/paper/2011.10675/) (2020)
- [How Image Degradations Affect Deep CNN-based Face Recognition?](https://mlsanity.com/paper/1608.05246/) (2016)

## Built on by (most cited first, 7 of 7)

- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)
- [RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning](https://mlsanity.com/paper/2308.14899/) (2023)
- [Enhancing object detection robustness: A synthetic and natural perturbation approach](https://mlsanity.com/paper/2304.10622/) (2023)
- [Causality-Driven Audits of Model Robustness](https://mlsanity.com/paper/2410.23494/) (2024)
- [A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy](https://mlsanity.com/paper/2508.14079/) (2025)
- [Fine-Grained ImageNet Classification in the Wild](https://mlsanity.com/paper/2303.02400/) (2023)
- [A Causal Framework for Aligning Image Quality Metrics and Deep Neural Network Robustness](https://mlsanity.com/paper/2503.02797/) (2025)

## Abstract (arXiv)

Synthetic corruptions gathered into a benchmark are frequently used to measure neural network robustness to distribution shifts. However, robustness to synthetic corruption benchmarks is not always predictive of robustness to distribution shifts encountered in real-world applications. In this paper, we propose a methodology to build synthetic corruption benchmarks that make robustness estimations more correlated with robustness to real-world distribution shifts. Using the overlapping criterion, we split synthetic corruptions into categories that help to better understand neural network robustness. Based on these categories, we identify three relevant parameters to take into account when constructing a corruption benchmark that are the (1) number of represented categories, (2) their relative balance in terms of size and, (3) the size of the considered benchmark. In doing so, we build new synthetic corruption selections that are more predictive of robustness to natural corruptions than existing synthetic corruption benchmarks.