# Open-Vocabulary Object Detection via Scene Graph Discovery arXiv: https://arxiv.org/abs/2307.03339 ยท HTML: https://mlsanity.com/paper/2307.03339/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 49 of its 66 references that are in the map. - First posted on arXiv 7 Jul 2023. ## Builds on (most cited first, 30 of 49) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015) - [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019) - [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021) - [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021) - [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021) - [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022) - [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017) - [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020) - [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016) - [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022) - [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022) - [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022) - [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018) - [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017) - [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022) - [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022) - [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021) - [PromptDet: Towards Open-vocabulary Detection using Uncurated Images](https://mlsanity.com/paper/2203.16513/) (2022) - [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021) ## Built on by (most cited first, 7 of 7) - [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023) - [Generative Region-Language Pretraining for Open-Ended Object Detection](https://mlsanity.com/paper/2403.10191/) (2024) - [3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding](https://mlsanity.com/paper/2412.18450/) (2024) - [Weakly Supervised Open-Vocabulary Object Detection](https://mlsanity.com/paper/2312.12437/) (2023) - [Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection](https://mlsanity.com/paper/2403.14270/) (2024) - [Open-Vocabulary Object Detectors: Robustness Challenges under Distribution Shifts](https://mlsanity.com/paper/2405.14874/) (2024) - [DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2310.01393/) (2023) ## Abstract (arXiv) In recent years, open-vocabulary (OV) object detection has attracted increasing research attention. Unlike traditional detection, which only recognizes fixed-category objects, OV detection aims to detect objects in an open category set. Previous works often leverage vision-language (VL) training data (e.g., referring grounding data) to recognize OV objects. However, they only use pairs of nouns and individual objects in VL data, while these data usually contain much more information, such as scene graphs, which are also crucial for OV detection. In this paper, we propose a novel Scene-Graph-Based Discovery Network (SGDN) that exploits scene graph cues for OV detection. Firstly, a scene-graph-based decoder (SGDecoder) including sparse scene-graph-guided attention (SSGA) is presented. It captures scene graphs and leverages them to discover OV objects. Secondly, we propose scene-graph-based prediction (SGPred), where we build a scene-graph-based offset regression (SGOR) mechanism to enable mutual enhancement between scene graph extraction and object localization. Thirdly, we design a cross-modal learning mechanism in SGPred. It takes scene graphs as bridges to improve the consistency between cross-modal embeddings for OV object classification. Experiments on COCO and LVIS demonstrate the effectiveness of our approach. Moreover, we show the ability of our model for OV scene graph detection, while previous OV scene graph generation methods cannot tackle this task.