# Towards Open-Set Object Detection and Discovery

arXiv: https://arxiv.org/abs/2204.05604 · HTML: https://mlsanity.com/paper/2204.05604/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [A Unified Objective for Novel Class Discovery](https://mlsanity.com/paper/2108.08536/) (2021)
- [Automatically Discovering and Learning New Visual Categories with Ranking Statistics](https://mlsanity.com/paper/2002.05714/) (2020)
- [Neighborhood Contrastive Learning for Novel Class Discovery](https://mlsanity.com/paper/2106.10731/) (2021)
- [AutoNovel: Automatically Discovering and Learning Novel Visual Categories](https://mlsanity.com/paper/2106.15252/) (2021)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)

## Built on by (most cited first, 30 of 44)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Goal-Driven Explainable Clustering via Language Descriptions](https://mlsanity.com/paper/2305.13749/) (2023)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [Novel Class Discovery: an Introduction and Key Concepts](https://mlsanity.com/paper/2302.12028/) (2023)
- [Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection](https://mlsanity.com/paper/2406.00510/) (2024)
- [AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset](https://mlsanity.com/paper/2306.00612/) (2023)
- [MOVE: Unsupervised Movable Object Segmentation and Detection](https://mlsanity.com/paper/2210.07920/) (2022)
- [Happy: A Debiased Learning Framework for Continual Generalized Category Discovery](https://mlsanity.com/paper/2410.06535/) (2024)
- [Curved Geometric Networks for Visual Anomaly Recognition](https://mlsanity.com/paper/2208.01188/) (2022)
- [PointCaM: Cut-and-Mix for Open-Set Point Cloud Learning](https://mlsanity.com/paper/2212.02011/) (2022)
- [Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling](https://mlsanity.com/paper/2403.01053/) (2024)
- [Towards Generalized Few-Shot Open-Set Object Detection](https://mlsanity.com/paper/2210.15996/) (2022)
- [Mitigating Label Noise on Graph via Topological Sample Selection](https://mlsanity.com/paper/2403.01942/) (2024)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [DINO-Detect: A Simple yet Effective Framework for Blur-Robust AI-Generated Image Detection](https://mlsanity.com/paper/2511.12511/) (2025)
- [SmokeBench: Evaluating Multimodal Large Language Models for Wildfire Smoke Detection](https://mlsanity.com/paper/2512.11215/) (2025)
- [USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model](https://mlsanity.com/paper/2306.02275/) (2023)
- [NOMAD: A Natural, Occluded, Multi-scale Aerial Dataset, for Emergency Response Scenarios](https://mlsanity.com/paper/2309.09518/) (2023)
- [Spoof Diarization: "What Spoofed When" in Partially Spoofed Audio](https://mlsanity.com/paper/2406.07816/) (2024)
- [MedDCR: Learning to Design Agentic Workflows for Medical Coding](https://mlsanity.com/paper/2511.13361/) (2025)
- [A Practical Approach to Novel Class Discovery in Tabular Data](https://mlsanity.com/paper/2311.05440/) (2023)
- [OpenNet: Incremental Learning for Autonomous Driving Object Detection with Balanced Loss](https://mlsanity.com/paper/2311.14939/) (2023)
- [PANDAS: Prototype-based Novel Class Discovery and Detection](https://mlsanity.com/paper/2402.17420/) (2024)
- [UADet: A Remarkably Simple Yet Effective Uncertainty-Aware Open-Set Object Detection Framework](https://mlsanity.com/paper/2412.09229/) (2024)
- [Text Classification Under Class Distribution Shift: A Survey](https://mlsanity.com/paper/2502.12965/) (2025)
- [Rethinking Open-Set Object Detection: Issues, a New Formulation, and Taxonomy](https://mlsanity.com/paper/2207.09775/) (2022)
- [Efficient Object Detection in Autonomous Driving using Spiking Neural Networks: Performance, Energy Consumption Analysis, and Insights into Open-set Object Discovery](https://mlsanity.com/paper/2312.07466/) (2023)
- [O1O: Grouping of Known Classes to Identify Unknown Objects as Odd-One-Out](https://mlsanity.com/paper/2410.07514/) (2024)
- [OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration](https://mlsanity.com/paper/2507.19870/) (2025)
- [FSP-DETR: Few-Shot Prototypical Parasitic Ova Detection](https://mlsanity.com/paper/2510.09583/) (2025)

## Abstract (arXiv)

With the human pursuit of knowledge, open-set object detection (OSOD) has been designed to identify unknown objects in a dynamic world. However, an issue with the current setting is that all the predicted unknown objects share the same category as "unknown", which require incremental learning via a human-in-the-loop approach to label novel classes. In order to address this problem, we present a new task, namely Open-Set Object Detection and Discovery (OSODD). This new task aims to extend the ability of open-set object detectors to further discover the categories of unknown objects based on their visual appearance without human effort. We propose a two-stage method that first uses an open-set object detector to predict both known and unknown objects. Then, we study the representation of predicted objects in an unsupervised manner and discover new categories from the set of unknown objects. With this method, a detector is able to detect objects belonging to known classes and define novel categories for objects of unknown classes with minimal supervision. We show the performance of our model on the MS-COCO dataset under a thorough evaluation protocol. We hope that our work will promote further research towards a more robust real-world detection system.