# Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation

arXiv: https://arxiv.org/abs/2007.01951 · HTML: https://mlsanity.com/paper/2007.01951/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 30 of 51)

- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Multi-View Transformer for 3D Visual Grounding](https://mlsanity.com/paper/2204.02174/) (2022)
- [TubeDETR: Spatio-Temporal Video Grounding with Transformers](https://mlsanity.com/paper/2203.16434/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [Referring Image Segmentation Using Text Supervision](https://mlsanity.com/paper/2308.14575/) (2023)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning](https://mlsanity.com/paper/2207.01328/) (2022)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [Shatter and Gather: Learning Referring Image Segmentation with Text Supervision](https://mlsanity.com/paper/2308.15512/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Adapting CLIP For Phrase Localization Without Further Training](https://mlsanity.com/paper/2204.03647/) (2022)
- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [Distilling Coarse-to-Fine Semantic Matching Knowledge for Weakly Supervised 3D Visual Grounding](https://mlsanity.com/paper/2307.09267/) (2023)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding](https://mlsanity.com/paper/2502.20678/) (2025)
- [Region-aware Knowledge Distillation for Efficient Image-to-Image Translation](https://mlsanity.com/paper/2205.12451/) (2022)
- [Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2501.17053/) (2025)
- [Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships](https://mlsanity.com/paper/2203.14260/) (2022)
- [WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2505.18686/) (2025)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [Who are you referring to? Coreference resolution in image narrations](https://mlsanity.com/paper/2211.14563/) (2022)
- [Making a Bird AI Expert Work for You and Me](https://mlsanity.com/paper/2112.02747/) (2021)
- [Cycle-Consistency Learning for Captioning and Grounding](https://mlsanity.com/paper/2312.15162/) (2023)
- [Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration](https://mlsanity.com/paper/2412.08939/) (2024)

## Abstract (arXiv)

Weakly supervised phrase grounding aims at learning region-phrase correspondences using only image-sentence pairs. A major challenge thus lies in the missing links between image regions and sentence phrases during training. To address this challenge, we leverage a generic object detector at training time, and propose a contrastive learning framework that accounts for both region-phrase and image-sentence matching. Our core innovation is the learning of a region-phrase score function, based on which an image-sentence score function is further constructed. Importantly, our region-phrase score function is learned by distilling from soft matching scores between the detected object names and candidate phrases within an image-sentence pair, while the image-sentence score function is supervised by ground-truth image-sentence pairs. The design of such score functions removes the need of object detection at test time, thereby significantly reducing the inference cost. Without bells and whistles, our approach achieves state-of-the-art results on visual phrase grounding, surpassing previous methods that require expensive object detectors at test time.