mlsanityData to 7 Oct 2026

Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts

arXiv 2107.12052 ↗First posted 26 Jul 2021cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Synthetic corruptions gathered into a benchmark are frequently used to measure neural network robustness to distribution shifts. However, robustness to synthetic corruption benchmarks is not always predictive of robustness to distribution shifts encountered in real-world applications. In this paper, we propose a methodology to build synthetic corruption benchmarks that make robustness estimations more correlated with robustness to real-world distribution shifts. Using the overlapping criterion, we split synthetic corruptions into categories that help to better understand neural network robustness. Based on these categories, we identify three relevant parameters to take into account when constructing a corruption benchmark that are the (1) number of represented categories, (2) their relative balance in terms of size and, (3) the size of the considered benchmark. In doing so, we build new synthetic corruption selections that are more predictive of robustness to natural corruptions than existing synthetic corruption benchmarks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 26 Jul 2021.

Builds on

32 of 39 references.

References · 32

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  12. 2018Recognition in Terra Incognitacited by 655
20 more
  1. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  2. 2018Albumentations: fast and flexible image augmentationscited by 558
  3. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  4. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  5. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  6. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  7. 2019Adversarial Examples Improve Image Recognitioncited by 283
  8. 2019Fast AutoAugmentcited by 243
  9. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  10. 2019Benchmarking the Robustness of Semantic Segmentation Modelscited by 111
  11. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  12. 2020MoPro: Webly Supervised Learning with Momentum Prototypescited by 66
  13. 2017CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognitioncited by 32
  14. 2019Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?cited by 25
  15. 2019How is Gaze Influenced by Image Transformations? Dataset and Modelcited by 24
  16. 2020Identifying Statistical Bias in Dataset Replicationcited by 24
  17. 2020An Effective Anti-Aliasing Approach for Residual Networkscited by 23
  18. 2016How Image Degradations Affect Deep CNN-based Face Recognition?cited by 20
  19. 2016PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognitioncited by 18
  20. 2021Using the Overlapping Score to Improve Corruption Benchmarkscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 2.

7 papers.

  1. 20223D Common Corruptions and Data Augmentationcited by 120
  2. 2023RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learningcited by 6
  3. 2023Enhancing object detection robustness: A synthetic and natural perturbation approachcited by 3
  4. 2024Causality-Driven Audits of Model Robustnesscited by 2
  5. 2025A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategycited by 1
  6. 2023Fine-Grained ImageNet Classification in the Wildcited by 0
  7. 2025A Causal Framework for Aligning Image Quality Metrics and Deep Neural Network Robustnesscited by 0