# Learning to Detect Every Thing in an Open World

arXiv: https://arxiv.org/abs/2112.01698 · HTML: https://mlsanity.com/paper/2112.01698/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Classification-Reconstruction Learning for Open-Set Recognition](https://mlsanity.com/paper/1812.04246/) (2018)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)

## Built on by (most cited first, 30 of 37)

- [PROB: Probabilistic Objectness for Open World Object Detection](https://mlsanity.com/paper/2212.01424/) (2022)
- [Open-vocabulary Panoptic Segmentation with Embedding Modulation](https://mlsanity.com/paper/2303.11324/) (2023)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization](https://mlsanity.com/paper/2210.07571/) (2022)
- [OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation](https://mlsanity.com/paper/2403.19580/) (2024)
- [Open-Set Semi-Supervised Object Detection](https://mlsanity.com/paper/2208.13722/) (2022)
- [SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learning](https://mlsanity.com/paper/2308.06531/) (2023)
- [Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detection](https://mlsanity.com/paper/2306.14291/) (2023)
- [3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation](https://mlsanity.com/paper/2310.03534/) (2023)
- [GOOD: Exploring Geometric Cues for Detecting Objects in an Open World](https://mlsanity.com/paper/2212.11720/) (2022)
- [Towards Generalized Few-Shot Open-Set Object Detection](https://mlsanity.com/paper/2210.15996/) (2022)
- [Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision](https://mlsanity.com/paper/2303.05503/) (2023)
- [Region-Aware Metric Learning for Open World Semantic Segmentation via Meta-Channel Aggregation](https://mlsanity.com/paper/2205.08083/) (2022)
- [3D Indoor Instance Segmentation in an Open-World](https://mlsanity.com/paper/2309.14338/) (2023)
- [Unsupervised Recognition of Unknown Objects for Open-World Object Detection](https://mlsanity.com/paper/2308.16527/) (2023)
- [Video Instance Segmentation in an Open-World](https://mlsanity.com/paper/2304.01200/) (2023)
- [Exploring Transformers for Open-world Instance Segmentation](https://mlsanity.com/paper/2308.04206/) (2023)
- [NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds](https://mlsanity.com/paper/2206.11736/) (2022)
- [OpenInst: A Simple Query-Based Method for Open-World Instance Segmentation](https://mlsanity.com/paper/2303.15859/) (2023)
- [Impact of Pseudo Depth on Open World Object Segmentation with Minimal User Guidance](https://mlsanity.com/paper/2304.05716/) (2023)
- [USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model](https://mlsanity.com/paper/2306.02275/) (2023)
- [Detecting Every Object from Events](https://mlsanity.com/paper/2404.05285/) (2024)
- [SOS: Segment Object System for Open-World Instance Segmentation With Object Priors](https://mlsanity.com/paper/2409.14627/) (2024)
- [From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoning](https://mlsanity.com/paper/2502.05843/) (2025)
- [Segment Any Events with Language](https://mlsanity.com/paper/2601.23159/) (2026)
- [Tunable Hybrid Proposal Networks for the Open World](https://mlsanity.com/paper/2208.11050/) (2022)
- [Addressing the Challenges of Open-World Object Detection](https://mlsanity.com/paper/2303.14930/) (2023)
- [OpenNet: Incremental Learning for Autonomous Driving Object Detection with Balanced Loss](https://mlsanity.com/paper/2311.14939/) (2023)
- [Generalized Class Discovery in Instance Segmentation](https://mlsanity.com/paper/2502.08149/) (2025)
- [Towards 3D Objectness Learning in an Open World](https://mlsanity.com/paper/2510.17686/) (2025)

## Abstract (arXiv)

Many open-world applications require the detection of novel objects, yet state-of-the-art object detection and instance segmentation networks do not excel at this task. The key issue lies in their assumption that regions without any annotations should be suppressed as negatives, which teaches the model to treat the unannotated objects as background. To address this issue, we propose a simple yet surprisingly powerful data augmentation and training scheme we call Learning to Detect Every Thing (LDET). To avoid suppressing hidden objects, background objects that are visible but unlabeled, we paste annotated objects on a background image sampled from a small region of the original image. Since training solely on such synthetically-augmented images suffers from domain shift, we decouple the training into two parts: 1) training the region classification and regression head on augmented images, and 2)~training the mask heads on original images. In this way, a model does not learn to classify hidden objects as background while generalizing well to real images. LDET leads to significant improvements on many datasets in the open-world instance segmentation task, outperforming baselines on cross-category generalization on COCO, as well as cross-dataset evaluation on UVO and Cityscapes.