# Learning Unsupervised Visual Grounding Through Semantic Self-Supervision

arXiv: https://arxiv.org/abs/1803.06506 · HTML: https://mlsanity.com/paper/1803.06506/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2018.

## Built on by (most cited first, 10 of 10)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Improved Visual Grounding through Self-Consistent Explanations](https://mlsanity.com/paper/2312.04554/) (2023)
- [Detector-Free Weakly Supervised Grounding by Separation](https://mlsanity.com/paper/2104.09829/) (2021)
- [What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs](https://mlsanity.com/paper/2206.09358/) (2022)
- [Box-based Refinement for Weakly Supervised and Unsupervised Localization Tasks](https://mlsanity.com/paper/2309.03874/) (2023)
- [Encoder-Decoder Based Long Short-Term Memory (LSTM) Model for Video Captioning](https://mlsanity.com/paper/2401.02052/) (2023)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement](https://mlsanity.com/paper/2305.10913/) (2023)

## Abstract (arXiv)

Localizing natural language phrases in images is a challenging problem that requires joint understanding of both the textual and visual modalities. In the unsupervised setting, lack of supervisory signals exacerbate this difficulty. In this paper, we propose a novel framework for unsupervised visual grounding which uses concept learning as a proxy task to obtain self-supervision. The simple intuition behind this idea is to encourage the model to localize to regions which can explain some semantic property in the data, in our case, the property being the presence of a concept in a set of images. We present thorough quantitative and qualitative experiments to demonstrate the efficacy of our approach and show a 5.6% improvement over the current state of the art on Visual Genome dataset, a 5.8% improvement on the ReferItGame dataset and comparable to state-of-art performance on the Flickr30k dataset.