# COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts

arXiv: https://arxiv.org/abs/2307.12730 · HTML: https://mlsanity.com/paper/2307.12730/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 87 of its 103 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 30 of 87)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)

## Built on by (most cited first, 30 of 34)

- [LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models](https://mlsanity.com/paper/2501.18954/) (2025)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection](https://mlsanity.com/paper/2404.09216/) (2024)
- [BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models](https://mlsanity.com/paper/2312.02896/) (2023)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [WeDetect: Fast Open-Vocabulary Object Detection as Retrieval](https://mlsanity.com/paper/2512.12309/) (2025)
- [Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models](https://mlsanity.com/paper/2410.19635/) (2024)
- [Benchmarking Segmentation Models with Mask-Preserved Attribute Editing](https://mlsanity.com/paper/2403.01231/) (2024)
- [Open-Vocabulary Object Detectors: Robustness Challenges under Distribution Shifts](https://mlsanity.com/paper/2405.14874/) (2024)
- [MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspective](https://mlsanity.com/paper/2507.19131/) (2025)
- [SOOD-ImageNet: a Large-Scale Dataset for Semantic Out-Of-Distribution Image Classification and Semantic Segmentation](https://mlsanity.com/paper/2409.01109/) (2024)
- [Causality-Driven Audits of Model Robustness](https://mlsanity.com/paper/2410.23494/) (2024)
- [Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery](https://mlsanity.com/paper/2503.19202/) (2025)
- [From COCO to COCO-FP: A Deep Dive into Background False Positives for COCO Detectors](https://mlsanity.com/paper/2409.07907/) (2024)
- [COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts](https://mlsanity.com/paper/2504.10158/) (2025)
- [Can We Challenge Open-Vocabulary Object Detectors with Generated Content in Street Scenes?](https://mlsanity.com/paper/2506.23751/) (2025)
- [WiSE-OD: Benchmarking Robustness in Infrared Object Detection](https://mlsanity.com/paper/2507.18925/) (2025)
- [A Causally Grounded Taxonomy for Image Degradation Robustness Evaluation](https://mlsanity.com/paper/2605.15906/) (2026)
- [Semantic Robustness Probing via Inpainting: An Interactive Tool for Safety-Critical Object Detection](https://mlsanity.com/paper/2605.27155/) (2026)
- [Robust Onion: Peeling Open Vocab Object Detectors Under Noise](https://mlsanity.com/paper/2606.26734/) (2026)
- [Identification of Fine-grained Systematic Errors via Controlled Scene Generation](https://mlsanity.com/paper/2404.07045/) (2024)
- [On the Robustness of Human-Object Interaction Detection against Distribution Shift](https://mlsanity.com/paper/2506.18021/) (2025)
- [GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks](https://mlsanity.com/paper/2512.24592/) (2025)
- [Towards Robust Cross-Dataset Object Detection Generalization under Domain Specificity](https://mlsanity.com/paper/2601.09497/) (2026)
- [ObjEmbed: Towards Universal Multimodal Object Embeddings](https://mlsanity.com/paper/2602.01753/) (2026)
- [Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method](https://mlsanity.com/paper/2603.27556/) (2026)
- [Monte Carlo Stochastic Depth for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/2604.12719/) (2026)
- [UAV-OVO: Out-of-Viewpoint Generalization in UAV Action Recognition](https://mlsanity.com/paper/2605.25615/) (2026)
- [DOME: Learning Transferable Domain Variables from Sparse Supervision for Test-Time Adaptation](https://mlsanity.com/paper/2606.07646/) (2026)
- [ContextShift: A Controlled Benchmark for Context Dependence in Object Detection](https://mlsanity.com/paper/2606.09495/) (2026)

## Abstract (arXiv)

Practical object detection application can lose its effectiveness on image inputs with natural distribution shifts. This problem leads the research community to pay more attention on the robustness of detectors under Out-Of-Distribution (OOD) inputs. Existing works construct datasets to benchmark the detector's OOD robustness for a specific application scenario, e.g., Autonomous Driving. However, these datasets lack universality and are hard to benchmark general detectors built on common tasks such as COCO. To give a more comprehensive robustness assessment, we introduce COCO-O(ut-of-distribution), a test dataset based on COCO with 6 types of natural distribution shifts. COCO-O has a large distribution gap with training data and results in a significant 55.7% relative performance drop on a Faster R-CNN detector. We leverage COCO-O to conduct experiments on more than 100 modern object detectors to investigate if their improvements are credible or just over-fitting to the COCO test set. Unfortunately, most classic detectors in early years do not exhibit strong OOD generalization. We further study the robustness effect on recent breakthroughs of detector's architecture design, augmentation and pre-training techniques. Some empirical findings are revealed: 1) Compared with detection head or neck, backbone is the most important part for robustness; 2) An end-to-end detection transformer design brings no enhancement, and may even reduce robustness; 3) Large-scale foundation models have made a great leap on robust object detection. We hope our COCO-O could provide a rich testbed for robustness study of object detection. The dataset will be available at https://github.com/alibaba/easyrobust/tree/main/benchmarks/coco_o.