# Object Detection as Probabilistic Set Prediction

arXiv: https://arxiv.org/abs/2203.07980 · HTML: https://mlsanity.com/paper/2203.07980/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [Probabilistic two-stage detection](https://mlsanity.com/paper/2103.07461/) (2021)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving](https://mlsanity.com/paper/1904.04620/) (2019)
- [Deep Set Prediction Networks](https://mlsanity.com/paper/1906.06565/) (2019)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Reliable training and estimation of variance networks](https://mlsanity.com/paper/1906.03260/) (2019)
- [Probabilistic Object Detection: Definition and Evaluation](https://mlsanity.com/paper/1811.10800/) (2018)
- [Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector](https://mlsanity.com/paper/1901.10609/) (2019)
- [Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors](https://mlsanity.com/paper/2101.05036/) (2021)
- [Poisson multi-Bernoulli mixture filter: direct derivation and implementation](https://mlsanity.com/paper/1703.04264/) (2017)
- [Localization Uncertainty Estimation for Anchor-Free Object Detection](https://mlsanity.com/paper/2006.15607/) (2020)
- [I Find Your Lack of Uncertainty in Computer Vision Disturbing](https://mlsanity.com/paper/2104.08188/) (2021)
- [Capturing Object Detection Uncertainty in Multi-Layer Grid Maps](https://mlsanity.com/paper/1901.11284/) (2019)
- [An Uncertainty-Aware Performance Measure for Multi-Object Tracking](https://mlsanity.com/paper/2108.04619/) (2021)
- [Scaling Data Association for Hypothesis-Oriented MHT](https://mlsanity.com/paper/1905.07466/) (2019)

## Built on by (most cited first, 3 of 3)

- [On permutation-invariant neural networks](https://mlsanity.com/paper/2403.17410/) (2024)
- [Label Dependencies-aware Set Prediction Networks for Multi-label Text Classification](https://mlsanity.com/paper/2304.07022/) (2023)
- [Hinge-Wasserstein: Estimating Multimodal Aleatoric Uncertainty in Regression Tasks](https://mlsanity.com/paper/2306.00560/) (2023)

## Abstract (arXiv)

Accurate uncertainty estimates are essential for deploying deep object detectors in safety-critical systems. The development and evaluation of probabilistic object detectors have been hindered by shortcomings in existing performance measures, which tend to involve arbitrary thresholds or limit the detector's choice of distributions. In this work, we propose to view object detection as a set prediction task where detectors predict the distribution over the set of objects. Using the negative log-likelihood for random finite sets, we present a proper scoring rule for evaluating and training probabilistic object detectors. The proposed method can be applied to existing probabilistic detectors, is free from thresholds, and enables fair comparison between architectures. Three different types of detectors are evaluated on the COCO dataset. Our results indicate that the training of existing detectors is optimized toward non-probabilistic metrics. We hope to encourage the development of new object detectors that can accurately estimate their own uncertainty. Code available at https://github.com/georghess/pmb-nll.