# Empirical Upper Bound in Object Detection and More

arXiv: https://arxiv.org/abs/1911.12451 · HTML: https://mlsanity.com/paper/1911.12451/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2019.

## Built on by (most cited first, 12 of 12)

- [Scale-aware Automatic Augmentation for Object Detection](https://mlsanity.com/paper/2103.17220/) (2021)
- [Towards Robust Adaptive Object Detection under Noisy Annotations](https://mlsanity.com/paper/2204.02620/) (2022)
- [TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentation](https://mlsanity.com/paper/2307.14611/) (2023)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations](https://mlsanity.com/paper/2409.09412/) (2024)
- [Box Re-Ranking: Unsupervised False Positive Suppression for Domain Adaptive Pedestrian Detection](https://mlsanity.com/paper/2102.00595/) (2021)
- [Robust Detection Outcome: A Metric for Pathology Detection in Medical Images](https://mlsanity.com/paper/2303.01920/) (2023)
- [TIVE: A Toolbox for Identifying Video Instance Segmentation Errors](https://mlsanity.com/paper/2210.08856/) (2022)
- [Sensitivity of Average Precision to Bounding Box Perturbations](https://mlsanity.com/paper/2206.10107/) (2022)
- [Complementary datasets to COCO for object detection](https://mlsanity.com/paper/2206.11473/) (2022)
- [Unsupervised Cross-domain Pulmonary Nodule Detection without Source Data](https://mlsanity.com/paper/2304.01085/) (2023)
- [Differences in Detection: Explainability Where it Matters](https://mlsanity.com/paper/2606.07503/) (2026)

## Abstract (arXiv)

Object detection remains as one of the most notorious open problems in computer vision. Despite large strides in accuracy in recent years, modern object detectors have started to saturate on popular benchmarks raising the question of how far we can reach with deep learning tools and tricks. Here, by employing 2 state-of-the-art object detection benchmarks, and analyzing more than 15 models over 4 large scale datasets, we I) carefully determine the upperbound in AP, which is 91.6% on VOC (test2007), 78.2% on COCO (val2017), and 58.9% on OpenImages V4 (validation), regardless of the IOU. These numbers are much better than the mAP of the best model1 (47.9% on VOC, and 46.9% on COCO; IOUs=.5:.95), II) characterize the sources of errors in object detectors, in a novel and intuitive way, and find that classification error (confusion with other classes and misses) explains the largest fraction of errors and weighs more than localization and duplicate errors, and III) analyze the invariance properties of models when surrounding context of an object is removed, when an object is placed in an incongruent background, and when images are blurred or flipped vertically. We find that models generate boxes on empty regions and that context is more important for detecting small objects than larger ones. Our work taps into the tight relationship between recognition and detection and offers insights for building better models.