mlsanityData to 7 Oct 2026

Read, look and detect: Bounding box annotation from image-caption pairs

arXiv 2306.06149 ↗First posted 9 Jun 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Various methods have been proposed to detect objects while reducing the cost of data annotation. For instance, weakly supervised object detection (WSOD) methods rely only on image-level annotations during training. Unfortunately, data annotation remains expensive since annotators must provide the categories describing the content of each image and labeling is restricted to a fixed set of categories. In this paper, we propose a method to locate and label objects in an image by using a form of weaker supervision: image-caption pairs. By leveraging recent advances in vision-language (VL) models and self-supervised vision transformers (ViTs), our method is able to perform phrase grounding and object detection in a weakly supervised manner. Our experiments demonstrate the effectiveness of our approach by achieving a 47.51% recall@1 score in phrase grounding on Flickr30k Entities and establishing a new state-of-the-art in object detection by achieving 21.1 mAP 50 and 10.5 mAP 50:95 on MS COCO when exclusively relying on image-caption pairs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

46 of 57 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Mask R-CNNcited by 7,259
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
34 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2021Grounded Language-Image Pre-trainingcited by 1,147
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  10. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  11. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  12. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 305
  13. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  14. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  15. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  16. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  17. 2015Weakly Supervised Deep Detection Networkscited by 183
  18. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
  19. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  20. 2018PCL: Proposal Cluster Learning for Weakly Supervised Object Detectioncited by 97
  21. 2020Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detectioncited by 80
  22. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labelscited by 71
  23. 2022Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectorscited by 69
  24. 2019WSOD^2: Learning Bottom-up and Top-down Objectness Distillation for Weakly-supervised Object Detectioncited by 51
  25. 2020Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detectioncited by 50
  26. 2016Weakly Supervised Cascaded Convolutional Networkscited by 47
  27. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  28. 2018Towards Human-Machine Cooperation: Self-supervised Sample Mining for Object Detectioncited by 32
  29. 2019Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detectioncited by 31
  30. 2020Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detectioncited by 29
  31. 2020Solving Missing-Annotation Object Detection with Background Recalibration Losscited by 27
  32. 2022Active Learning Strategies for Weakly-supervised Object Detectioncited by 17
  33. 2022SIOD: Single Instance Annotated Per Category Per Image for Object Detectioncited by 10
  34. 2017Image Captioning with Object Detection and Localizationcited by 2

Built on by

Not cited yet.