mlsanityData to 7 Oct 2026

Measuring Robustness to Natural Distribution Shifts in Image Classification

arXiv 2007.00644 ↗First posted 1 Jul 2020cs.LG cs.CV stat.MLcited by 452

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at https://modestyachts.github.io/imagenet-testbed/ .

Key facts. Cited by 452 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 75; 2022: 82; 2023: 105; 2024: 66; 2025: 70; 2026: 54.

452 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2021Domain Generalization: A Surveycited by 760
  9. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  12. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
38 more
  1. 2023ClimaX: A foundation model for weather and climatecited by 325
  2. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  3. 2023Tool Learning with Foundation Modelscited by 302
  4. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  5. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  6. 2021Partial success in closing the gap between human and machine visioncited by 185
  7. 2022Parameter-free Online Test-time Adaptationcited by 184
  8. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  9. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  10. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  11. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  12. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  13. 2022Causality Inspired Representation Learning for Domain Generalizationcited by 136
  14. 20223D Common Corruptions and Data Augmentationcited by 120
  15. 2021Predicting with Confidence on Unseen Distributionscited by 116
  16. 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
  17. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  18. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  19. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  20. 2021Model-Based Domain Generalizationcited by 103
  21. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  22. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  23. 2021On Interaction Between Augmentations and Corruptions in Natural Corruption Robustnesscited by 98
  24. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  25. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  26. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  27. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  28. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  29. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  30. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  31. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  32. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  33. 2022The Fallacy of AI Functionalitycited by 88
  34. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  35. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  36. 2021Learning Neural Network Subspacescited by 82
  37. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 78
  38. 2024Mission Critical -- Satellite Data is a Distinct Modality in Machine Learningcited by 76

50 most cited of 452 All 452 (JSON)