# Detector-Free Weakly Supervised Grounding by Separation

arXiv: https://arxiv.org/abs/2104.09829 · HTML: https://mlsanity.com/paper/2104.09829/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 82 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)

## Built on by (most cited first, 16 of 16)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Described Object Detection: Liberating Object Detection with Flexible Expressions](https://mlsanity.com/paper/2307.12813/) (2023)
- [Referring Image Segmentation Using Text Supervision](https://mlsanity.com/paper/2308.14575/) (2023)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Improved Visual Grounding through Self-Consistent Explanations](https://mlsanity.com/paper/2312.04554/) (2023)
- [What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs](https://mlsanity.com/paper/2206.09358/) (2022)
- [What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions](https://mlsanity.com/paper/2303.16990/) (2023)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning](https://mlsanity.com/paper/2510.06243/) (2025)
- [Box-based Refinement for Weakly Supervised and Unsupervised Localization Tasks](https://mlsanity.com/paper/2309.03874/) (2023)
- [Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM](https://mlsanity.com/paper/2404.19128/) (2024)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement](https://mlsanity.com/paper/2305.10913/) (2023)
- [AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2508.03201/) (2025)
- [SCoRD: Subject-Conditional Relation Detection with Text-Augmented Data](https://mlsanity.com/paper/2308.12910/) (2023)

## Abstract (arXiv)

Nowadays, there is an abundance of data involving images and surrounding free-form text weakly corresponding to those images. Weakly Supervised phrase-Grounding (WSG) deals with the task of using this data to learn to localize (or to ground) arbitrary text phrases in images without any additional annotations. However, most recent SotA methods for WSG assume the existence of a pre-trained object detector, relying on it to produce the ROIs for localization. In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector. We directly learn everything from the images and associated free-form text pairs, thus potentially gaining an advantage on the categories unsupported by the detector. The key idea behind our proposed Grounding by Separation (GbS) method is synthesizing `text to image-regions' associations by random alpha-blending of arbitrary image pairs and using the corresponding texts of the pair as conditions to recover the alpha map from the blended image via a segmentation network. At test time, this allows using the query phrase as a condition for a non-blended query image, thus interpreting the test image as a composition of a region corresponding to the phrase and the complement region. Using this approach we demonstrate a significant accuracy improvement, of up to 8.5% over previous DF-WSG SotA, for a range of benchmarks including Flickr30K, Visual Genome, and ReferIt, as well as a significant complementary improvement (above 7%) over the detector-based approaches for WSG.