# What's in the Black Box? The False Negative Mechanisms Inside Object Detectors

arXiv: https://arxiv.org/abs/2203.07662 · HTML: https://mlsanity.com/paper/2203.07662/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Getting to Know Low-light Images with The Exclusively Dark Dataset](https://mlsanity.com/paper/1805.11227/) (2018)
- [Imbalance Problems in Object Detection: A Review](https://mlsanity.com/paper/1909.00169/) (2019)
- [TIDE: A General Toolbox for Identifying Object Detection Errors](https://mlsanity.com/paper/2008.08115/) (2020)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAM](https://mlsanity.com/paper/1804.04011/) (2018)
- [WiderPerson: A Diverse Dataset for Dense Pedestrian Detection in the Wild](https://mlsanity.com/paper/1909.12118/) (2019)
- [Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Cars](https://mlsanity.com/paper/1707.00051/) (2017)
- [Uncertainty for Identifying Open-Set Errors in Visual Object Detection](https://mlsanity.com/paper/2104.01328/) (2021)
- [Heterogeneous Ground and Air Platforms, Homogeneous Sensing: Team CSIRO Data61's Approach to the DARPA Subterranean Challenge](https://mlsanity.com/paper/2104.09053/) (2021)
- [Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors](https://mlsanity.com/paper/1903.06391/) (2019)
- [Are we done with object recognition? The iCub robot's perspective](https://mlsanity.com/paper/1709.09882/) (2017)
- [The CSIRO Crown-of-Thorn Starfish Detection Dataset](https://mlsanity.com/paper/2111.14311/) (2021)

## Built on by (most cited first, 7 of 7)

- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [Monitoring of Perception Systems: Deterministic, Probabilistic, and Learning-based Fault Detection and Identification](https://mlsanity.com/paper/2205.10906/) (2022)
- [Revealing Similar Semantics Inside CNNs: An Interpretable Concept-based Comparison of Feature Spaces](https://mlsanity.com/paper/2305.07663/) (2023)
- [Are All Marine Species Created Equal? Performance Disparities in Underwater Object Detection](https://mlsanity.com/paper/2508.18729/) (2025)
- [Enhancing Your Trained DETRs with Box Refinement](https://mlsanity.com/paper/2307.11828/) (2023)
- [Perception Simplex: Verifiable Collision Avoidance in Autonomous Vehicles Amidst Obstacle Detection Faults](https://mlsanity.com/paper/2209.01710/) (2022)
- [SKU-Patch: Towards Efficient Instance Segmentation for Unseen Objects in Auto-Store](https://mlsanity.com/paper/2311.04645/) (2023)

## Abstract (arXiv)

In object detection, false negatives arise when a detector fails to detect a target object. To understand why object detectors produce false negatives, we identify five 'false negative mechanisms', where each mechanism describes how a specific component inside the detector architecture failed. Focusing on two-stage and one-stage anchor-box object detector architectures, we introduce a framework for quantifying these false negative mechanisms. Using this framework, we investigate why Faster R-CNN and RetinaNet fail to detect objects in benchmark vision datasets and robotics datasets. We show that a detector's false negative mechanisms differ significantly between computer vision benchmark datasets and robotics deployment scenarios. This has implications for the translation of object detectors developed for benchmark datasets to robotics applications. Code is publicly available at https://github.com/csiro-robotics/fn_mechanisms