# Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild

arXiv: https://arxiv.org/abs/2203.03800 · HTML: https://mlsanity.com/paper/2203.03800/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 73 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)

## Built on by (most cited first, 30 of 70)

- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Masked Discrimination for Self-Supervised Learning on Point Clouds](https://mlsanity.com/paper/2203.11183/) (2022)
- [Non-Parametric Outlier Synthesis](https://mlsanity.com/paper/2303.02966/) (2023)
- [POEM: Out-of-Distribution Detection with Posterior Sampling](https://mlsanity.com/paper/2206.13687/) (2022)
- [LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning](https://mlsanity.com/paper/2306.01293/) (2023)
- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [Dream the Impossible: Outlier Imagination with Diffusion Models](https://mlsanity.com/paper/2309.13415/) (2023)
- [Negative Label Guided OOD Detection with Pretrained Vision-Language Models](https://mlsanity.com/paper/2403.20078/) (2024)
- [How Does Unlabeled Data Provably Help Out-of-Distribution Detection?](https://mlsanity.com/paper/2402.03502/) (2024)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [A Survey on Out-of-Distribution Detection in NLP](https://mlsanity.com/paper/2305.03236/) (2023)
- [Unknown Sniffer for Object Detection: Don't Turn a Blind Eye to Unknown Objects](https://mlsanity.com/paper/2303.13769/) (2023)
- [LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models](https://mlsanity.com/paper/2407.08966/) (2024)
- [Secure On-Device Video OOD Detection Without Backpropagation](https://mlsanity.com/paper/2503.06166/) (2025)
- [Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection](https://mlsanity.com/paper/2302.07106/) (2023)
- [AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models](https://mlsanity.com/paper/2410.20149/) (2024)
- [On the use of Mahalanobis distance for out-of-distribution detection with neural networks for medical imaging](https://mlsanity.com/paper/2309.01488/) (2023)
- [How to Overcome Curse-of-Dimensionality for Out-of-Distribution Detection?](https://mlsanity.com/paper/2312.14452/) (2023)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [RoDLA: Benchmarking the Robustness of Document Layout Analysis Models](https://mlsanity.com/paper/2403.14442/) (2024)
- [Open World Object Detection: A Survey](https://mlsanity.com/paper/2410.11301/) (2024)
- [Teacher-Student Network for 3D Point Cloud Anomaly Detection with Few Normal Samples](https://mlsanity.com/paper/2210.17258/) (2022)
- [LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition](https://mlsanity.com/paper/2309.12780/) (2023)
- [Can OOD Object Detectors Learn from Foundation Models?](https://mlsanity.com/paper/2409.05162/) (2024)
- [Model-free Test Time Adaptation for Out-Of-Distribution Detection](https://mlsanity.com/paper/2311.16420/) (2023)

## Abstract (arXiv)

Building reliable object detectors that can detect out-of-distribution (OOD) objects is critical yet underexplored. One of the key challenges is that models lack supervision signals from unknown data, producing overconfident predictions on OOD objects. We propose a new unknown-aware object detection framework through Spatial-Temporal Unknown Distillation (STUD), which distills unknown objects from videos in the wild and meaningfully regularizes the model's decision boundary. STUD first identifies the unknown candidate object proposals in the spatial dimension, and then aggregates the candidates across multiple video frames to form a diverse set of unknown objects near the decision boundary. Alongside, we employ an energy-based uncertainty regularization loss, which contrastively shapes the uncertainty space between the in-distribution and distilled unknown objects. STUD establishes the state-of-the-art performance on OOD detection tasks for object detection, reducing the FPR95 score by over 10% compared to the previous best method. Code is available at https://github.com/deeplearning-wisc/stud.