# Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration

arXiv: https://arxiv.org/abs/2307.00934 · HTML: https://mlsanity.com/paper/2307.00934/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 83 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)

## Built on by (most cited first, 19 of 19)

- [MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection](https://mlsanity.com/paper/2309.14976/) (2023)
- [Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding](https://mlsanity.com/paper/2403.17010/) (2024)
- [On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines](https://mlsanity.com/paper/2405.20459/) (2024)
- [Beyond Classification: Definition and Density-based Estimation of Calibration in Object Detection](https://mlsanity.com/paper/2312.06645/) (2023)
- [U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences](https://mlsanity.com/paper/2512.02982/) (2025)
- [MaxEnt Loss: Constrained Maximum Entropy for Calibration under Out-of-Distribution Shift](https://mlsanity.com/paper/2310.17159/) (2023)
- [ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning](https://mlsanity.com/paper/2409.18026/) (2024)
- [DrIFT: Autonomous Drone Dataset with Integrated Real and Synthetic Data, Flexible Views, and Transformed Domains](https://mlsanity.com/paper/2412.04789/) (2024)
- [Uncertainty-Aware Autonomous Vehicles: Predicting the Road Ahead](https://mlsanity.com/paper/2510.22680/) (2025)
- [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025)
- [Uncertainty Quantification in Detection Transformers: Object-Level Calibration and Image-Level Reliability](https://mlsanity.com/paper/2412.01782/) (2024)
- [A Theoretical and Practical Framework for Evaluating Uncertainty Calibration in Object Detection](https://mlsanity.com/paper/2309.00464/) (2023)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [Self-Aware Object Detection via Degradation Manifolds](https://mlsanity.com/paper/2602.18394/) (2026)
- [Query2Uncertainty: Robust Uncertainty Quantification and Calibration for 3D Object Detection under Distribution Shift](https://mlsanity.com/paper/2605.05328/) (2026)
- [A Causally Grounded Taxonomy for Image Degradation Robustness Evaluation](https://mlsanity.com/paper/2605.15906/) (2026)
- [UTrack: Multi-Object Tracking with Uncertain Detections](https://mlsanity.com/paper/2408.17098/) (2024)
- [Calibrating Probabilistic Object Detectors with Annotator Disagreement](https://mlsanity.com/paper/2605.24722/) (2026)
- [Rethinking Detection Calibration: A Coordinate and Direction Perspective](https://mlsanity.com/paper/2607.29040/) (2026)

## Abstract (arXiv)

The current approach for testing the robustness of object detectors suffers from serious deficiencies such as improper methods of performing out-of-distribution detection and using calibration metrics which do not consider both localisation and classification quality. In this work, we address these issues, and introduce the Self-Aware Object Detection (SAOD) task, a unified testing framework which respects and adheres to the challenges that object detectors face in safety-critical environments such as autonomous driving. Specifically, the SAOD task requires an object detector to be: robust to domain shift; obtain reliable uncertainty estimates for the entire scene; and provide calibrated confidence scores for the detections. We extensively use our framework, which introduces novel metrics and large scale test datasets, to test numerous object detectors in two different use-cases, allowing us to highlight critical insights into their robustness performance. Finally, we introduce a simple baseline for the SAOD task, enabling researchers to benchmark future proposed methods and move towards robust object detectors which are fit for purpose. Code is available at https://github.com/fiveai/saod