# Solving Instance Detection from an Open-World Perspective

arXiv: https://arxiv.org/abs/2503.00359 · HTML: https://mlsanity.com/paper/2503.00359/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction](https://mlsanity.com/paper/2304.06714/) (2023)
- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [Finetune like you pretrain: Improved finetuning of zero-shot vision models](https://mlsanity.com/paper/2212.00638/) (2022)
- [OpenGAN: Open-Set Recognition via Open Data Generation](https://mlsanity.com/paper/2104.02939/) (2021)
- [Multiple Instance Detection Network with Online Instance Classifier Refinement](https://mlsanity.com/paper/1704.00138/) (2017)
- [Adversarial AutoAugment](https://mlsanity.com/paper/1912.11188/) (2019)

## Built on by (most cited first, 6 of 6)

- [Dare to Plagiarize? Plagiarized Painting Recognition and Retrieval](https://mlsanity.com/paper/2506.23132/) (2025)
- [Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2602.00813/) (2026)
- [Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective](https://mlsanity.com/paper/2512.10244/) (2025)
- [From Local Matches to Global Masks: Template-Guided Instance Detection and Segmentation in Open-World Scenes](https://mlsanity.com/paper/2603.03577/) (2026)
- [FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs](https://mlsanity.com/paper/2606.04282/) (2026)
- [Task-driven Processing with Coarse-to-Fine Glimpse-based Active Perception](https://mlsanity.com/paper/2609.09025/) (2026)

## Abstract (arXiv)

Instance detection (InsDet) aims to localize specific object instances within a novel scene imagery based on given visual references. Technically, it requires proposal detection to identify all possible object instances, followed by instance-level matching to pinpoint the ones of interest. Its open-world nature supports its broad applications from robotics to AR/VR but also presents significant challenges: methods must generalize to unknown testing data distributions because (1) the testing scene imagery is unseen during training, and (2) there are domain gaps between visual references and detected proposals. Existing methods tackle these challenges by synthesizing diverse training examples or utilizing off-the-shelf foundation models (FMs). However, they only partially capitalize the available open-world information. In contrast, we approach InsDet from an Open-World perspective, introducing our method IDOW. We find that, while pretrained FMs yield high recall in instance detection, they are not specifically optimized for instance-level feature matching. Therefore, we adapt pretrained FMs for improved instance-level matching using open-world data. Our approach incorporates metric learning along with novel data augmentations, which sample distractors as negative examples and synthesize novel-view instances to enrich the visual references. Extensive experiments demonstrate that our method significantly outperforms prior works, achieving >10 AP over previous results on two recently released challenging benchmark datasets in both conventional and novel instance detection settings.