# Gradient-Based Quantification of Epistemic Uncertainty for Deep Object Detectors

arXiv: https://arxiv.org/abs/2107.04517 · HTML: https://mlsanity.com/paper/2107.04517/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 66 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving](https://mlsanity.com/paper/1904.04620/) (2019)
- [Multivariate Confidence Calibration for Object Detection](https://mlsanity.com/paper/2004.13546/) (2020)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)
- [Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection](https://mlsanity.com/paper/1809.06006/) (2018)

## Built on by (most cited first, 16 of 16)

- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection](https://mlsanity.com/paper/2302.07106/) (2023)
- [Pixel-wise Gradient Uncertainty for Convolutional Neural Networks applied to Out-of-Distribution Segmentation](https://mlsanity.com/paper/2303.06920/) (2023)
- [Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data](https://mlsanity.com/paper/2112.12833/) (2021)
- [DrIFT: Autonomous Drone Dataset with Integrated Real and Synthetic Data, Flexible Views, and Transformed Domains](https://mlsanity.com/paper/2412.04789/) (2024)
- [Revisiting Gradient-based Uncertainty for Monocular Depth Estimation](https://mlsanity.com/paper/2502.05964/) (2025)
- [LMD: Light-weight Prediction Quality Estimation for Object Detection in Lidar Point Clouds](https://mlsanity.com/paper/2306.07835/) (2023)
- [Conformal Object Detection by Sequential Risk Control](https://mlsanity.com/paper/2505.24038/) (2025)
- [Can We Challenge Open-Vocabulary Object Detectors with Generated Content in Street Scenes?](https://mlsanity.com/paper/2506.23751/) (2025)
- [Uncertainty Quantification and Resource-Demanding Computer Vision Applications of Deep Learning](https://mlsanity.com/paper/2205.14917/) (2022)
- [GIT: Detecting Uncertainty, Out-Of-Distribution and Adversarial Samples using Gradients and Invariance Transformations](https://mlsanity.com/paper/2307.02672/) (2023)
- [Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks](https://mlsanity.com/paper/2409.11373/) (2024)
- [Towards Reliable Detection of Empty Space: Conditional Marked Point Processes for Object Detection](https://mlsanity.com/paper/2506.21486/) (2025)
- [Uncertainty-Aware Subset Selection for Robust Visual Explainability under Distribution Shifts](https://mlsanity.com/paper/2512.08445/) (2025)
- [CUPID: A Plug-in Framework for Joint Aleatoric and Epistemic Uncertainty Estimation with a Single Model](https://mlsanity.com/paper/2603.10745/) (2026)
- [Hard to See, Hard to Label: Generative and Symbolic Acquisition for Subtle Visual Phenomena](https://mlsanity.com/paper/2604.22990/) (2026)

## Abstract (arXiv)

The vast majority of uncertainty quantification methods for deep object detectors such as variational inference are based on the network output. Here, we study gradient-based epistemic uncertainty metrics for deep object detectors to obtain reliable confidence estimates. We show that they contain predictive information and that they capture information orthogonal to that of common, output-based uncertainty estimation methods like Monte-Carlo dropout and deep ensembles. To this end, we use meta classification and meta regression to produce confidence estimates using gradient metrics and other baselines for uncertainty quantification which are in principle applicable to any object detection architecture. Specifically, we employ false positive detection and prediction of localization quality to investigate uncertainty content of our metrics and compute the calibration errors of meta classifiers. Moreover, we use them as a post-processing filter mechanism to the object detection pipeline and compare object detection performance. Our results show that gradient-based uncertainty is itself on par with output-based methods across different detectors and datasets. More significantly, combined meta classifiers based on gradient and output-based metrics outperform the standalone models. Based on this result, we conclude that gradient uncertainty adds orthogonal information to output-based methods. This suggests that variational inference may be supplemented by gradient-based uncertainty to obtain improved confidence measures, contributing to down-stream applications of deep object detectors and improving their probabilistic reliability.