# Language Adaptive Weight Generation for Multi-task Visual Grounding

arXiv: https://arxiv.org/abs/2306.04652 · HTML: https://mlsanity.com/paper/2306.04652/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [CondConv: Conditionally Parameterized Convolutions for Efficient Inference](https://mlsanity.com/paper/1904.04971/) (2019)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)

## Built on by (most cited first, 30 of 35)

- [CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities](https://mlsanity.com/paper/2408.13239/) (2024)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling](https://mlsanity.com/paper/2410.08021/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Grounding](https://mlsanity.com/paper/2408.01120/) (2024)
- [VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders](https://mlsanity.com/paper/2309.01141/) (2023)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints](https://mlsanity.com/paper/2501.06710/) (2025)
- [InstructDET: Diversifying Referring Object Detection with Generalized Instructions](https://mlsanity.com/paper/2310.05136/) (2023)
- [SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding](https://mlsanity.com/paper/2407.03200/) (2024)
- [DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergy](https://mlsanity.com/paper/2507.01738/) (2025)
- [PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discrimination](https://mlsanity.com/paper/2509.04833/) (2025)
- [WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2505.18686/) (2025)
- [SwimVG: Step-wise Multimodal Fusion and Adaption for Visual Grounding](https://mlsanity.com/paper/2502.16786/) (2025)
- [Improving Generalized Visual Grounding with Instance-aware Joint Learning](https://mlsanity.com/paper/2509.13747/) (2025)
- [Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understanding](https://mlsanity.com/paper/2404.08590/) (2024)
- [Cycle-Consistency Learning for Captioning and Grounding](https://mlsanity.com/paper/2312.15162/) (2023)
- [MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension](https://mlsanity.com/paper/2409.13609/) (2024)
- [UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization](https://mlsanity.com/paper/2404.03179/) (2024)
- [GVDIFF: Grounded Text-to-Video Generation with Diffusion Models](https://mlsanity.com/paper/2407.01921/) (2024)
- [CLASH: Complementary Learning with Neural Architecture Search for Gait Recognition](https://mlsanity.com/paper/2407.03632/) (2024)
- [Referencing Where to Focus: Improving VisualGrounding with Referential Query](https://mlsanity.com/paper/2412.19155/) (2024)
- [SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation](https://mlsanity.com/paper/2603.18086/) (2026)
- [Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding](https://mlsanity.com/paper/2412.00684/) (2024)
- [A Simple and Better Baseline for Visual Grounding](https://mlsanity.com/paper/2510.10587/) (2025)
- [BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Grounding](https://mlsanity.com/paper/2601.01526/) (2026)
- [Make Graph-based Referring Expression Comprehension Great Again through Expression-guided Dynamic Gating and Regression](https://mlsanity.com/paper/2409.03385/) (2024)
- [Phrase-Instance Alignment for Generalized Referring Segmentation](https://mlsanity.com/paper/2411.15087/) (2024)
- [Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2509.06291/) (2025)
- [RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding](https://mlsanity.com/paper/2602.00504/) (2026)

## Abstract (arXiv)

Although the impressive performance in visual grounding, the prevailing approaches usually exploit the visual backbone in a passive way, i.e., the visual backbone extracts features with fixed weights without expression-related hints. The passive perception may lead to mismatches (e.g., redundant and missing), limiting further performance improvement. Ideally, the visual backbone should actively extract visual features since the expressions already provide the blueprint of desired visual features. The active perception can take expressions as priors to extract relevant visual features, which can effectively alleviate the mismatches. Inspired by this, we propose an active perception Visual Grounding framework based on Language Adaptive Weights, called VG-LAW. The visual backbone serves as an expression-specific feature extractor through dynamic weights generated for various expressions. Benefiting from the specific and relevant visual features extracted from the language-aware visual backbone, VG-LAW does not require additional modules for cross-modal interaction. Along with a neat multi-task head, VG-LAW can be competent in referring expression comprehension and segmentation jointly. Extensive experiments on four representative datasets, i.e., RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, validate the effectiveness of the proposed framework and demonstrate state-of-the-art performance.