# Uncertainty for Identifying Open-Set Errors in Visual Object Detection

arXiv: https://arxiv.org/abs/2104.01328 · HTML: https://mlsanity.com/paper/2104.01328/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 3 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [Classification-Reconstruction Learning for Open-Set Recognition](https://mlsanity.com/paper/1812.04246/) (2018)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving](https://mlsanity.com/paper/1903.08701/) (2019)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving](https://mlsanity.com/paper/1904.04620/) (2019)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection](https://mlsanity.com/paper/1804.05132/) (2018)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Probabilistic Object Detection: Definition and Evaluation](https://mlsanity.com/paper/1811.10800/) (2018)
- [Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection](https://mlsanity.com/paper/1809.06006/) (2018)
- [Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors](https://mlsanity.com/paper/2101.05036/) (2021)
- [Uncertainty Estimation in One-Stage Object Detection](https://mlsanity.com/paper/1905.10296/) (2019)
- [Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving](https://mlsanity.com/paper/1909.12358/) (2019)
- [Localization Uncertainty Estimation for Anchor-Free Object Detection](https://mlsanity.com/paper/2006.15607/) (2020)
- [Leveraging Uncertainties for Deep Multi-modal Object Detection in Autonomous Driving](https://mlsanity.com/paper/2002.00216/) (2020)
- [Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2003.03644/) (2020)
- [Calibrating Uncertainties in Object Localization Task](https://mlsanity.com/paper/1811.11210/) (2018)
- [Towards Better Performance and More Explainable Uncertainty for 3D Object Detection of Autonomous Vehicles](https://mlsanity.com/paper/2006.12015/) (2020)
- [Are we done with object recognition? The iCub robot's perspective](https://mlsanity.com/paper/1709.09882/) (2017)

## Built on by (most cited first, 26 of 26)

- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [Objects in Semantic Topology](https://mlsanity.com/paper/2110.02687/) (2021)
- [Open-Set Semi-Supervised Object Detection](https://mlsanity.com/paper/2208.13722/) (2022)
- [Out-of-Distribution Detection for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2209.14435/) (2022)
- [Hyperdimensional Feature Fusion for Out-Of-Distribution Detection](https://mlsanity.com/paper/2112.05341/) (2021)
- [Bayesian Deep Learning for Affordance Segmentation in images](https://mlsanity.com/paper/2303.00871/) (2023)
- [Open-Set Recognition in the Age of Vision-Language Models](https://mlsanity.com/paper/2403.16528/) (2024)
- [Towards Generalized Few-Shot Open-Set Object Detection](https://mlsanity.com/paper/2210.15996/) (2022)
- [What's in the Black Box? The False Negative Mechanisms Inside Object Detectors](https://mlsanity.com/paper/2203.07662/) (2022)
- [Open-Set Object Detection Using Classification-free Object Proposal and Instance-level Contrastive Learning](https://mlsanity.com/paper/2211.11530/) (2022)
- [Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning](https://mlsanity.com/paper/2311.06481/) (2023)
- [Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations](https://mlsanity.com/paper/2403.01172/) (2024)
- [USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model](https://mlsanity.com/paper/2306.02275/) (2023)
- [Uncertainty-Aware Lidar Place Recognition in Novel Environments](https://mlsanity.com/paper/2210.01361/) (2022)
- [Addressing the Challenges of Open-World Object Detection](https://mlsanity.com/paper/2303.14930/) (2023)
- [UADet: A Remarkably Simple Yet Effective Uncertainty-Aware Open-Set Object Detection Framework](https://mlsanity.com/paper/2412.09229/) (2024)
- [OoDDINO:A Multi-level Framework for Anomaly Segmentation on Complex Road Scenes](https://mlsanity.com/paper/2507.01455/) (2025)
- [Generalized Open-World Semi-Supervised Object Detection](https://mlsanity.com/paper/2307.15710/) (2023)
- [O1O: Grouping of Known Classes to Identify Unknown Objects as Odd-One-Out](https://mlsanity.com/paper/2410.07514/) (2024)
- [Model-Agnostic Open-Set Air-to-Air Visual Object Detection for Reliable UAV Perception](https://mlsanity.com/paper/2509.09297/) (2025)
- [Towards Robust Object Detection: Bayesian RetinaNet for Homoscedastic Aleatoric Uncertainty Modeling](https://mlsanity.com/paper/2108.00784/) (2021)
- [Detecting Out-of-distribution Objects Using Neuron Activation Patterns](https://mlsanity.com/paper/2307.16433/) (2023)
- [Wasserstein Distance-based Expansion of Low-Density Latent Regions for Unknown Class Detection](https://mlsanity.com/paper/2401.05594/) (2024)
- [YOLOOC: YOLO-based Open-Class Incremental Object Detection with Novel Class Discovery](https://mlsanity.com/paper/2404.00257/) (2024)
- [OSR-ViT: A Simple and Modular Framework for Open-Set Object Detection and Discovery](https://mlsanity.com/paper/2404.10865/) (2024)
- [Three-Way Open-Set Detection for Robust Autonomous Navigation](https://mlsanity.com/paper/2511.15343/) (2025)

## Abstract (arXiv)

Deployed into an open world, object detectors are prone to open-set errors, false positive detections of object classes not present in the training dataset. We propose GMM-Det, a real-time method for extracting epistemic uncertainty from object detectors to identify and reject open-set errors. GMM-Det trains the detector to produce a structured logit space that is modelled with class-specific Gaussian Mixture Models. At test time, open-set errors are identified by their low log-probability under all Gaussian Mixture Models. We test two common detector architectures, Faster R-CNN and RetinaNet, across three varied datasets spanning robotics and computer vision. Our results show that GMM-Det consistently outperforms existing uncertainty techniques for identifying and rejecting open-set detections, especially at the low-error-rate operating point required for safety-critical applications. GMM-Det maintains object detection performance, and introduces only minimal computational overhead. We also introduce a methodology for converting existing object detection datasets into specific open-set datasets to evaluate open-set performance in object detection.