# Towards Open World Object Detection

arXiv: https://arxiv.org/abs/2103.02603 · HTML: https://mlsanity.com/paper/2103.02603/

## Key facts

- Cited by 297 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 63 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)

## Built on by (most cited first, 30 of 297)

- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [OW-DETR: Open-world Detection Transformer](https://mlsanity.com/paper/2112.01513/) (2021)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)
- [CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.07724/) (2022)
- [Class-agnostic Object Detection with Multi-modal Transformer](https://mlsanity.com/paper/2111.11430/) (2021)
- [UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning](https://mlsanity.com/paper/2203.14542/) (2022)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation](https://mlsanity.com/paper/2204.02136/) (2022)
- [Open Vocabulary Object Detection with Pseudo Bounding-Box Labels](https://mlsanity.com/paper/2111.09452/) (2021)
- [Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild](https://mlsanity.com/paper/2203.03800/) (2022)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [Continual Detection Transformer for Incremental Object Detection](https://mlsanity.com/paper/2304.03110/) (2023)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [PROB: Probabilistic Objectness for Open World Object Detection](https://mlsanity.com/paper/2212.01424/) (2022)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [A Large-scale Multiple-objective Method for Black-box Attack against Object Detection](https://mlsanity.com/paper/2209.07790/) (2022)
- [OVTrack: Open-Vocabulary Multiple Object Tracking](https://mlsanity.com/paper/2304.08408/) (2023)
- [A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation](https://mlsanity.com/paper/2402.04087/) (2024)
- [Deep Metric Learning for Open World Semantic Segmentation](https://mlsanity.com/paper/2108.04562/) (2021)
- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [Towards Open-Set Object Detection and Discovery](https://mlsanity.com/paper/2204.05604/) (2022)
- [ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cues](https://mlsanity.com/paper/2201.06696/) (2022)
- [Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity](https://mlsanity.com/paper/2204.06107/) (2022)

## Abstract (arXiv)

Humans have a natural instinct to identify unknown object instances in their environments. The intrinsic curiosity about these unknown instances aids in learning about them, when the corresponding knowledge is eventually available. This motivates us to propose a novel computer vision problem called: `Open World Object Detection', where a model is tasked to: 1) identify objects that have not been introduced to it as `unknown', without explicit supervision to do so, and 2) incrementally learn these identified unknown categories without forgetting previously learned classes, when the corresponding labels are progressively received. We formulate the problem, introduce a strong evaluation protocol and provide a novel solution, which we call ORE: Open World Object Detector, based on contrastive clustering and energy based unknown identification. Our experimental evaluation and ablation studies analyze the efficacy of ORE in achieving Open World objectives. As an interesting by-product, we find that identifying and characterizing unknown instances helps to reduce confusion in an incremental object detection setting, where we achieve state-of-the-art performance, with no extra methodological effort. We hope that our work will attract further research into this newly identified, yet crucial research direction.