mlsanityData to 7 Oct 2026

Contrastive Learning for Weakly Supervised Phrase Grounding

arXiv 2006.09920 ↗First posted 17 Jun 2020cs.CV cs.CL cs.LGcited by 102

Not on the map. Why?

Abstract (arXiv)

Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a ~10% absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of 5.7% to achieve 76.7% accuracy on Flickr30K Entities benchmark.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 23; 2023: 25; 2024: 11; 2025: 14; 2026: 3.

102 papers.

Sort:

By area:

  1. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  2. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  3. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  4. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  5. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  6. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  7. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  8. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  9. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  10. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  11. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  12. 2024POP-3D: Open-Vocabulary 3D Occupancy Prediction from Imagescited by 52
38 more
  1. 2024Toward Visual Grounding: A Surveycited by 52
  2. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  3. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentationcited by 40
  4. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  5. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  6. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  7. 2023Referring Image Segmentation Using Text Supervisioncited by 35
  8. 2022Pseudo-Q: Generating Pseudo Language Queries for Visual Groundingcited by 33
  9. 2021Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learningcited by 30
  10. 2021Deep Matching Prior: Test-Time Optimization for Dense Correspondencecited by 30
  11. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  12. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  13. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  14. 2023Shatter and Gather: Learning Referring Image Segmentation with Text Supervisioncited by 25
  15. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  16. 2021Extending CLIP for Category-to-image Retrieval in E-commercecited by 24
  17. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanationscited by 24
  18. 2022Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 22
  19. 2024NetTrack: Tracking Highly Dynamic Objects with a Netcited by 22
  20. 2025Multimodal Referring Segmentation: A Surveycited by 22
  21. 2021Improving Pneumonia Localization via Cross-Attention on Medical Images and Reportscited by 21
  22. 2022Weakly-supervised segmentation of referring expressionscited by 21
  23. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  24. 2021Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learningcited by 20
  25. 2022Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generationcited by 20
  26. 2025Referring to Any Personcited by 20
  27. 2021Joint Multimedia Event Extraction from Video and Articlecited by 19
  28. 2021Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Visioncited by 18
  29. 2021Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videoscited by 18
  30. 2023Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Groundingcited by 18
  31. 2023Improved Visual Grounding through Self-Consistent Explanationscited by 17
  32. 2021Detector-Free Weakly Supervised Grounding by Separationcited by 16
  33. 2021Who's Waldo? Linking People Across Text and Imagescited by 16
  34. 2022RepsNet: Combining Vision with Language for Automated Medical Reportscited by 14
  35. 2021Weakly-Supervised Video Object Grounding via Causal Interventioncited by 13
  36. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  37. 2023Object Recognition as Next Token Predictioncited by 12
  38. 2022Curriculum Learning for Data-Efficient Vision-Language Alignmentcited by 11

50 most cited of 102 All 102 (JSON)