# Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection

arXiv: https://arxiv.org/abs/1907.10164 · HTML: https://mlsanity.com/paper/1907.10164/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2019.

## Built on by (most cited first, 30 of 31)

- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [Learning Object-Language Alignments for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2211.14843/) (2022)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models](https://mlsanity.com/paper/2305.19595/) (2023)
- [Open Vocabulary Object Detection with Pseudo Bounding-Box Labels](https://mlsanity.com/paper/2111.09452/) (2021)
- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)
- [Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining](https://mlsanity.com/paper/2107.14572/) (2021)
- [Linguistic Structures as Weak Supervision for Visual Scene Graph Generation](https://mlsanity.com/paper/2105.13994/) (2021)
- [Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation](https://mlsanity.com/paper/2301.00805/) (2023)
- [Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization](https://mlsanity.com/paper/2206.11134/) (2022)
- [Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning](https://mlsanity.com/paper/2311.11646/) (2023)
- [Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations](https://mlsanity.com/paper/2303.16891/) (2023)
- [ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2312.07266/) (2023)
- [FETA: Towards Specializing Foundation Models for Expert Task Applications](https://mlsanity.com/paper/2209.03648/) (2022)
- [Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection](https://mlsanity.com/paper/2310.12152/) (2023)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [Bridging Gap between Image Pixels and Semantics via Supervision: A Survey](https://mlsanity.com/paper/2107.13757/) (2021)
- [Boosting Weakly Supervised Object Detection using Fusion and Priors from Hallucinated Depth](https://mlsanity.com/paper/2303.10937/) (2023)
- [Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptation](https://mlsanity.com/paper/2411.02057/) (2024)
- [OVIS: Open-Vocabulary Visual Instance Search via Visual-Semantic Aligned Representation Learning](https://mlsanity.com/paper/2108.03704/) (2021)
- [Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval](https://mlsanity.com/paper/2206.08842/) (2022)
- [Weakly Supervised Visual-Auditory Fixation Prediction with Multigranularity Perception](https://mlsanity.com/paper/2112.13697/) (2021)
- [Semi-Supervised Domain Generalization for Object Detection via Language-Guided Feature Alignment](https://mlsanity.com/paper/2309.13525/) (2023)
- [Weakly-Supervised Action Detection Guided by Audio Narration](https://mlsanity.com/paper/2205.05895/) (2022)
- [VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection](https://mlsanity.com/paper/2303.09608/) (2023)
- [Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection](https://mlsanity.com/paper/2303.10093/) (2023)

## Abstract (arXiv)

Learning to localize and name object instances is a fundamental problem in vision, but state-of-the-art approaches rely on expensive bounding box supervision. While weakly supervised detection (WSOD) methods relax the need for boxes to that of image-level annotations, even cheaper supervision is naturally available in the form of unstructured textual descriptions that users may freely provide when uploading image content. However, straightforward approaches to using such data for WSOD wastefully discard captions that do not exactly match object names. Instead, we show how to squeeze the most information out of these captions by training a text-only classifier that generalizes beyond dataset boundaries. Our discovery provides an opportunity for learning detection models from noisy but more abundant and freely-available caption data. We also validate our model on three classic object detection benchmarks and achieve state-of-the-art WSOD performance. Our code is available at https://github.com/yekeren/Cap2Det.