mlsanityData to 7 Oct 2026

OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images

arXiv 2111.14341 ↗First posted 29 Nov 2021cs.CV cs.AIcited by 46

Not on the map. Why?

Abstract (arXiv)

Enhancing the robustness of vision algorithms in real-world scenarios is challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors. We introduce OOD-CV, a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions, and enables benchmarking models for image classification, object detection, and 3D pose estimation. In addition to this novel dataset, we contribute extensive experiments using popular baseline methods, which reveal that: 1. Some nuisance factors have a much stronger negative effect on the performance compared to others, also depending on the vision task. 2. Current approaches to enhance robustness have only marginal effects, and can even reduce robustness. 3. We do not observe significant differences between convolutional and transformer architectures. We believe our dataset provides a rich testbed to study robustness and will help push forward research in this area.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

39 of 62 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2019Searching for MobileNetV3cited by 1,984
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  12. 2019Natural Adversarial Examplescited by 1,401
27 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  2. 2020In Search of Lost Domain Generalizationcited by 1,042
  3. 2016Adversarial Machine Learning at Scalecited by 934
  4. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  5. 2020Object-Centric Learning with Slot Attentioncited by 865
  6. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  7. 2017Focal Loss for Dense Object Detectioncited by 749
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  9. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  10. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  11. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  12. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  13. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  14. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  15. 2021Are Transformers More Robust Than CNNs?cited by 187
  16. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  17. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  18. 2018Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimationcited by 92
  19. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  20. 2020Robust Object Detection under Occlusion with Context-Aware CompositionalNetscited by 54
  21. 2021Improving Contrastive Learning by Visualizing Feature Transformationcited by 44
  22. 2018StarMap for Category-Agnostic Keypoint and Viewpoint Estimationcited by 42
  23. 2016UnrealCV: Connecting Computer Vision to Unreal Enginecited by 41
  24. 2021NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimationcited by 41
  25. 2020Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition under Occlusioncited by 38
  26. 2017Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systemscited by 28
  27. 2020Distilling Visual Priors from Self-Supervised Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 17; 2025: 8; 2026: 4.

46 papers.

Sort:

By area:

  1. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  2. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  3. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  4. 2023COCO-O: A Benchmark for Object Detectors under Natural Distribution Shiftscited by 34
  5. 2024TimeX++: Learning Time-Series Explanations with Information Bottleneckcited by 32
  6. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 28
  7. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understandingcited by 25
  8. 2023Industrial Anomaly Detection with Domain Shift: A Real-world Dataset and Masked Multi-scale Reconstructioncited by 22
  9. 2024Explaining Time Series via Contrastive and Locally Sparse Perturbationscited by 22
  10. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 20
  11. 2023Generating Images with 3D Annotations Using Diffusion Modelscited by 18
  12. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethicscited by 14
34 more
  1. 2023OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 12
  2. 2024Rethinking VLMs and LLMs for Image Classificationcited by 12
  3. 2023RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learningcited by 6
  4. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?cited by 5
  5. 2023Benchmarking Robustness to Text-Guided Corruptionscited by 4
  6. 2023Neural Textured Deformable Meshes for Robust Analysis-by-Synthesiscited by 4
  7. 2023Unsupervised Camouflaged Object Segmentation as Domain Adaptationcited by 4
  8. 20233D Adversarial Augmentations for Robust Out-of-Domain Predictionscited by 4
  9. 2023GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shiftscited by 4
  10. 2023OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shiftcited by 4
  11. 2026Certified Circuits: Stability Guarantees for Mechanistic Circuitscited by 4
  12. 2024DomainVerse: A Benchmark Towards Real-World Distribution Shifts For Tuning-Free Adaptive Domain Generalizationcited by 3
  13. 2024A Bayesian Approach to OOD Robustness in Image Classificationcited by 3
  14. 2024WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusioncited by 3
  15. 2025Understanding and Mitigating Distribution Shifts For Machine Learning Force Fieldscited by 3
  16. 2026LychSim: A Controllable and Interactive Simulation Framework for Vision Researchcited by 3
  17. 2023Robust Category-Level 3D Pose Estimation from Synthetic Datacited by 2
  18. 2023Robustness May be More Brittle than We Think under Different Degrees of Distribution Shiftscited by 2
  19. 2024MEDDAP: Medical Dataset Enhancement via Diversified Augmentation Pipelinecited by 2
  20. 2024SOOD-ImageNet: a Large-Scale Dataset for Semantic Out-Of-Distribution Image Classification and Semantic Segmentationcited by 2
  21. 2024Causality-Driven Audits of Model Robustnesscited by 2
  22. 2024Cityscape-Adverse: Benchmarking Robustness of Semantic Segmentation with Realistic Scene Modifications via Diffusion-Based Image Editingcited by 2
  23. 2025Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoningcited by 2
  24. 2025Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagerycited by 2
  25. 2025CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shiftscited by 2
  26. 2024Latent Enhancing AutoEncoder for Occluded Image Classificationcited by 1
  27. 2024BelHouse3D: A Benchmark Dataset for Assessing Occlusion Robustness in 3D Point Cloud Semantic Segmentationcited by 1
  28. 2025COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shiftscited by 1
  29. 2025RO-Bench: Large-scale robustness evaluation of MLLMs with text-driven counterfactual videoscited by 1
  30. 2024Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shiftcited by 0
  31. 2025A Causal Framework for Aligning Image Quality Metrics and Deep Neural Network Robustnesscited by 0
  32. 2025DINeMo: Learning Neural Mesh Models with no 3D Annotationscited by 0
  33. 2026Robustness Emerges Early in Training Dynamics, but Is Not Preservedcited by 0
  34. 2026Towards A Unified Information Bottleneck Framework for Time Series Explanationscited by 0