# State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detection

arXiv: https://arxiv.org/abs/2511.18012 · HTML: https://mlsanity.com/paper/2511.18012/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 22 Nov 2025.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015)
- [Adversarial Complementary Learning for Weakly Supervised Object Localization](https://mlsanity.com/paper/1804.06962/) (2018)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [PromptDet: Towards Open-vocabulary Detection using Uncurated Images](https://mlsanity.com/paper/2203.16513/) (2022)
- [Probabilistic two-stage detection](https://mlsanity.com/paper/2103.07461/) (2021)
- [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023)
- [Multiple Instance Detection Network with Online Instance Classifier Refinement](https://mlsanity.com/paper/1704.00138/) (2017)
- [Learning Object-Language Alignments for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2211.14843/) (2022)
- [Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2305.07011/) (2023)
- [Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection](https://mlsanity.com/paper/2004.04725/) (2020)
- [C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1904.05647/) (2019)
- [Open Vocabulary Object Detection with Pseudo Bounding-Box Labels](https://mlsanity.com/paper/2111.09452/) (2021)
- [Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation](https://mlsanity.com/paper/2203.10593/) (2022)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models](https://mlsanity.com/paper/2501.18954/) (2025)
- [Multi-Modal Classifiers for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.05493/) (2023)

## Abstract (arXiv)

Open-Vocabulary Object Detection (OVOD) aims to generalize object recognition to novel categories, while Weakly Supervised OVOD (WS-OVOD) extends this by combining box-level annotations with image-level labels. Despite recent progress, two critical challenges persist in this setting. First, existing semantic prototypes, even when enriched by LLMs, are static and limited, failing to capture the rich intra-class visual variations induced by different object states (e.g., a cat's pose). Second, the standard pseudo-box generation introduces a semantic mismatch between visual region proposals (which contain context) and object-centric text embeddings. To tackle these issues, we introduce two complementary prototype enhancement strategies. To capture intra-class variations in appearance and state, we propose the State-Enhanced Semantic Prototypes (SESP), which generates state-aware textual descriptions (e.g., "a sleeping cat") to capture diverse object appearances, yielding more discriminative prototypes. Building on this, we further introduce Scene-Augmented Pseudo Prototypes (SAPP) to address the semantic mismatch. SAPP incorporates contextual semantics (e.g., "cat lying on sofa") and utilizes a soft alignment mechanism to promote contextually consistent visual-textual representations. By integrating SESP and SAPP, our method effectively enhances both the richness of semantic prototypes and the visual-textual alignment, achieving notable improvements.