# Knowledge Aided Consistency for Weakly Supervised Phrase Grounding

arXiv: https://arxiv.org/abs/1803.03879 · HTML: https://mlsanity.com/paper/1803.03879/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2018.

## Built on by (most cited first, 30 of 32)

- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus](https://mlsanity.com/paper/2207.01203/) (2022)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [Linguistic Structures as Weak Supervision for Visual Scene Graph Generation](https://mlsanity.com/paper/2105.13994/) (2021)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation](https://mlsanity.com/paper/2211.01412/) (2022)
- [Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding](https://mlsanity.com/paper/2106.04053/) (2021)
- [Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports](https://mlsanity.com/paper/2110.03094/) (2021)
- [Weakly-supervised segmentation of referring expressions](https://mlsanity.com/paper/2205.04725/) (2022)
- [Who's Waldo? Linking People Across Text and Images](https://mlsanity.com/paper/2108.07253/) (2021)
- [Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrases](https://mlsanity.com/paper/2207.01821/) (2022)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2501.17053/) (2025)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment](https://mlsanity.com/paper/2312.09625/) (2023)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching](https://mlsanity.com/paper/2105.05636/) (2021)
- [Distributed Attention for Grounded Image Captioning](https://mlsanity.com/paper/2108.01056/) (2021)
- [Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement](https://mlsanity.com/paper/2305.10913/) (2023)
- [A Better Loss for Visual-Textual Grounding](https://mlsanity.com/paper/2108.05308/) (2021)
- [AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2508.03201/) (2025)
- [How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding](https://mlsanity.com/paper/2402.19116/) (2024)
- [3DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignment](https://mlsanity.com/paper/2505.01809/) (2025)

## Abstract (arXiv)

Given a natural language query, a phrase grounding system aims to localize mentioned objects in an image. In weakly supervised scenario, mapping between image regions (i.e., proposals) and language is not available in the training set. Previous methods address this deficiency by training a grounding system via learning to reconstruct language information contained in input queries from predicted proposals. However, the optimization is solely guided by the reconstruction loss from the language modality, and ignores rich visual information contained in proposals and useful cues from external knowledge. In this paper, we explore the consistency contained in both visual and language modalities, and leverage complementary external knowledge to facilitate weakly supervised grounding. We propose a novel Knowledge Aided Consistency Network (KAC Net) which is optimized by reconstructing input query and proposal's information. To leverage complementary knowledge contained in the visual features, we introduce a Knowledge Based Pooling (KBP) gate to focus on query-related proposals. Experiments show that KAC Net provides a significant improvement on two popular datasets.