# Uncertainty Quantification in Detection Transformers: Object-Level Calibration and Image-Level Reliability

arXiv: https://arxiv.org/abs/2412.01782 · HTML: https://mlsanity.com/paper/2412.01782/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [Semantic Foggy Scene Understanding with Synthetic Data](https://mlsanity.com/paper/1708.07819/) (2017)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [Multivariate Confidence Calibration for Object Detection](https://mlsanity.com/paper/2004.13546/) (2020)
- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [Localization Recall Precision (LRP): A New Performance Metric for Object Detection](https://mlsanity.com/paper/1807.01696/) (2018)
- [Towards Improving Calibration in Object Detection Under Domain Shift](https://mlsanity.com/paper/2209.07601/) (2022)
- [Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration](https://mlsanity.com/paper/2307.00934/) (2023)
- [Multiclass Confidence and Localization Calibration for Object Detection](https://mlsanity.com/paper/2306.08271/) (2023)
- [One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks](https://mlsanity.com/paper/2011.10772/) (2020)
- [Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection](https://mlsanity.com/paper/2303.14404/) (2023)
- [Cal-DETR: Calibrated Detection Transformer](https://mlsanity.com/paper/2311.03570/) (2023)
- [On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines](https://mlsanity.com/paper/2405.20459/) (2024)
- [Domain Adaptive Object Detection via Balancing Between Self-Training and Adversarial Learning](https://mlsanity.com/paper/2311.04815/) (2023)
- [Tractable Uncertainty-Aware Meta-Learning](https://mlsanity.com/paper/2210.01881/) (2022)

## Built on by (most cited first, 2 of 2)

- [Know What You Don't Know: Uncertainty Calibration of Process Reward Models](https://mlsanity.com/paper/2506.09338/) (2025)
- [Calibrating Probabilistic Object Detectors with Annotator Disagreement](https://mlsanity.com/paper/2605.24722/) (2026)

## Abstract (arXiv)

DETR and its variants have emerged as promising architectures for object detection, offering an end-to-end prediction pipeline. In practice, however, DETRs generate hundreds of predictions that far outnumber the actual objects present in an image. This raises a critical question: which of these predictions could be trusted? This is particularly important for safety-critical applications, such as in autonomous vehicles. Addressing this concern, we provide empirical and theoretical evidence that predictions within the same image play distinct roles, resulting in varying reliability levels. Our analysis reveals that DETRs employ an optimal specialist strategy: one prediction per object is trained to be well-calibrated, while the remaining predictions are trained to suppress their foreground confidence to near zero, even when maintaining accurate localization. We show that this strategy emerges as the loss-minimizing solution to the Hungarian matching, fundamentally shaping DETRs' outputs. While selecting the well-calibrated predictions is ideal, they are unidentifiable at inference time. This means that any post-processing algorithm poses a risk of outputting a set of predictions with mixed calibration levels. Therefore, practical deployment necessitates a joint evaluation of both the model's calibration quality and the effectiveness of the post-processing algorithm. However, we demonstrate that existing metrics like average precision and expected calibration error are inadequate for this task. To address this issue, we further introduce Object-level Calibration Error (OCE): This object-centric design penalizes both retaining suppressed predictions and missed ground truth foreground objects, making OCE suitable for both evaluating models and identifying reliable prediction subsets. Finally, we present a post hoc uncertainty quantification framework that predicts per-image model accuracy.