# Context Disentangling and Prototype Inheriting for Robust Visual Grounding

arXiv: https://arxiv.org/abs/2312.11967 · HTML: https://mlsanity.com/paper/2312.11967/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 73 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 30 of 54)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning](https://mlsanity.com/paper/2104.03135/) (2021)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension](https://mlsanity.com/paper/1909.07072/) (2019)
- [Dynamic Graph Attention for Referring Expression Comprehension](https://mlsanity.com/paper/1909.08164/) (2019)
- [Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation](https://mlsanity.com/paper/2105.01839/) (2021)

## Built on by (most cited first, 23 of 23)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion](https://mlsanity.com/paper/2409.17531/) (2024)
- [Imp: Highly Capable Large Multimodal Models for Mobile Devices](https://mlsanity.com/paper/2405.12107/) (2024)
- [Visual Position Prompt for MLLM based Visual Grounding](https://mlsanity.com/paper/2503.15426/) (2025)
- [SwimVG: Step-wise Multimodal Fusion and Adaption for Visual Grounding](https://mlsanity.com/paper/2502.16786/) (2025)
- [ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery](https://mlsanity.com/paper/2604.01893/) (2026)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding](https://mlsanity.com/paper/2408.16314/) (2024)
- [MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension](https://mlsanity.com/paper/2409.13609/) (2024)
- [GIVEPose: Gradual Intra-class Variation Elimination for RGB-based Category-Level Object Pose Estimation](https://mlsanity.com/paper/2503.15110/) (2025)
- [FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing](https://mlsanity.com/paper/2505.01263/) (2025)
- [SynthDoc: Bilingual Documents Synthesis for Visual Document Understanding](https://mlsanity.com/paper/2408.14764/) (2024)
- [UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understanding](https://mlsanity.com/paper/2510.18262/) (2025)
- [Spatial Frequency Modulation for Semantic Segmentation](https://mlsanity.com/paper/2507.11893/) (2025)
- [A Recover-then-Discriminate Framework for Robust Anomaly Detection](https://mlsanity.com/paper/2406.04608/) (2024)
- [PathVG: A New Benchmark and Dataset for Pathology Visual Grounding](https://mlsanity.com/paper/2502.20869/) (2025)
- [Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles](https://mlsanity.com/paper/2512.03454/) (2025)
- [SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation](https://mlsanity.com/paper/2603.18086/) (2026)
- [Uncertainty-Aware Prototype Semantic Decoupling for Text-Based Person Search in Full Images](https://mlsanity.com/paper/2505.03567/) (2025)
- [AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2508.03201/) (2025)
- [MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance](https://mlsanity.com/paper/2509.09730/) (2025)
- [Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning](https://mlsanity.com/paper/2412.13543/) (2024)
- [Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2509.06291/) (2025)

## Abstract (arXiv)

Visual grounding (VG) aims to locate a specific target in an image based on a given language query. The discriminative information from context is important for distinguishing the target from other objects, particularly for the targets that have the same category as others. However, most previous methods underestimate such information. Moreover, they are usually designed for the standard scene (without any novel object), which limits their generalization to the open-vocabulary scene. In this paper, we propose a novel framework with context disentangling and prototype inheriting for robust visual grounding to handle both scenes. Specifically, the context disentangling disentangles the referent and context features, which achieves better discrimination between them. The prototype inheriting inherits the prototypes discovered from the disentangled visual features by a prototype bank to fully utilize the seen data, especially for the open-vocabulary scene. The fused features, obtained by leveraging Hadamard product on disentangled linguistic and visual features of prototypes to avoid sharp adjusting the importance between the two types of features, are then attached with a special token and feed to a vision Transformer encoder for bounding box regression. Extensive experiments are conducted on both standard and open-vocabulary scenes. The performance comparisons indicate that our method outperforms the state-of-the-art methods in both scenarios. {The code is available at https://github.com/WayneTomas/TransCP.