# OW-DETR: Open-world Detection Transformer

arXiv: https://arxiv.org/abs/2112.01513 · HTML: https://mlsanity.com/paper/2112.01513/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN](https://mlsanity.com/paper/2003.03901/) (2020)

## Built on by (most cited first, 30 of 129)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [PROB: Probabilistic Objectness for Open World Object Detection](https://mlsanity.com/paper/2212.01424/) (2022)
- [A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation](https://mlsanity.com/paper/2402.04087/) (2024)
- [CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection](https://mlsanity.com/paper/2310.02960/) (2023)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection](https://mlsanity.com/paper/2402.17323/) (2024)
- [UC-OWOD: Unknown-Classified Open World Object Detection](https://mlsanity.com/paper/2207.11455/) (2022)
- [Energy-based Latent Aligner for Incremental Learning](https://mlsanity.com/paper/2203.14952/) (2022)
- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection](https://mlsanity.com/paper/2307.12427/) (2023)
- [Revisiting Open World Object Detection](https://mlsanity.com/paper/2201.00471/) (2022)
- [End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2204.03541/) (2022)
- [CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detection](https://mlsanity.com/paper/2301.01970/) (2023)
- [CapDet: Unifying Dense Captioning and Open-World Detection Pretraining](https://mlsanity.com/paper/2303.02489/) (2023)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2303.15322/) (2023)
- [Random Boxes Are Open-world Object Detectors](https://mlsanity.com/paper/2307.08249/) (2023)
- [Modeling Missing Annotations for Incremental Learning in Object Detection](https://mlsanity.com/paper/2204.08766/) (2022)
- [Unknown Sniffer for Object Detection: Don't Turn a Blind Eye to Unknown Objects](https://mlsanity.com/paper/2303.13769/) (2023)
- [Towards Unsupervised Object Detection From LiDAR Point Clouds](https://mlsanity.com/paper/2311.02007/) (2023)
- [VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model](https://mlsanity.com/paper/2403.05346/) (2024)
- [OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision](https://mlsanity.com/paper/2405.17913/) (2024)
- [Exploring Incompatible Knowledge Transfer in Few-shot Image Generation](https://mlsanity.com/paper/2304.07574/) (2023)
- [Instruction-Guided Visual Masking](https://mlsanity.com/paper/2405.19783/) (2024)
- [Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2406.00830/) (2024)
- [Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object Detection](https://mlsanity.com/paper/2407.11499/) (2024)
- [Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts](https://mlsanity.com/paper/2410.05963/) (2024)

## Abstract (arXiv)

Open-world object detection (OWOD) is a challenging computer vision problem, where the task is to detect a known set of object categories while simultaneously identifying unknown objects. Additionally, the model must incrementally learn new classes that become known in the next training episodes. Distinct from standard object detection, the OWOD setting poses significant challenges for generating quality candidate proposals on potentially unknown objects, separating the unknown objects from the background and detecting diverse unknown objects. Here, we introduce a novel end-to-end transformer-based framework, OW-DETR, for open-world object detection. The proposed OW-DETR comprises three dedicated components namely, attention-driven pseudo-labeling, novelty classification and objectness scoring to explicitly address the aforementioned OWOD challenges. Our OW-DETR explicitly encodes multi-scale contextual information, possesses less inductive bias, enables knowledge transfer from known classes to the unknown class and can better discriminate between unknown objects and background. Comprehensive experiments are performed on two benchmarks: MS-COCO and PASCAL VOC. The extensive ablations reveal the merits of our proposed contributions. Further, our model outperforms the recently introduced OWOD approach, ORE, with absolute gains ranging from 1.8% to 3.3% in terms of unknown recall on MS-COCO. In the case of incremental object detection, OW-DETR outperforms the state-of-the-art for all settings on PASCAL VOC. Our code is available at https://github.com/akshitac8/OW-DETR.