# Bear the Query in Mind: Visual Grounding with Query-conditioned Convolution

arXiv: https://arxiv.org/abs/2206.09114 · HTML: https://mlsanity.com/paper/2206.09114/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 18 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension](https://mlsanity.com/paper/1909.07072/) (2019)
- [Dynamic Graph Attention for Referring Expression Comprehension](https://mlsanity.com/paper/1909.08164/) (2019)
- [Modeling Relationships in Referential Expressions with Compositional Modular Networks](https://mlsanity.com/paper/1611.09978/) (2016)
- [Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks](https://mlsanity.com/paper/1812.04794/) (2018)
- [Learning to Compose and Reason with Language Tree Structures for Visual Grounding](https://mlsanity.com/paper/1906.01784/) (2019)
- [Grounding Referring Expressions in Images by Variational Context](https://mlsanity.com/paper/1712.01892/) (2017)
- [Real-Time Referring Expression Comprehension by Single-Stage Grounding Network](https://mlsanity.com/paper/1812.03426/) (2018)
- [Learning Cross-modal Context Graph for Visual Grounding](https://mlsanity.com/paper/1911.09042/) (2019)
- [Visual Grounding with Transformers](https://mlsanity.com/paper/2105.04281/) (2021)
- [Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation](https://mlsanity.com/paper/2109.08478/) (2021)
- [Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters](https://mlsanity.com/paper/1907.02985/) (2019)

## Abstract (arXiv)

Visual grounding is a task that aims to locate a target object according to a natural language expression. As a multi-modal task, feature interaction between textual and visual inputs is vital. However, previous solutions mainly handle each modality independently before fusing them together, which does not take full advantage of relevant textual information while extracting visual features. To better leverage the textual-visual relationship in visual grounding, we propose a Query-conditioned Convolution Module (QCM) that extracts query-aware visual features by incorporating query information into the generation of convolutional kernels. With our proposed QCM, the downstream fusion module receives visual features that are more discriminative and focused on the desired object described in the expression, leading to more accurate predictions. Extensive experiments on three popular visual grounding datasets demonstrate that our method achieves state-of-the-art performance. In addition, the query-aware visual features are informative enough to achieve comparable performance to the latest methods when directly used for prediction without further multi-modal fusion.