# Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries

arXiv: https://arxiv.org/abs/1704.03944 · HTML: https://mlsanity.com/paper/1704.03944/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2017.

## Built on by (most cited first, 17 of 17)

- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation](https://mlsanity.com/paper/2305.17011/) (2023)
- [Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation](https://mlsanity.com/paper/2305.16318/) (2023)
- [Instance-Specific Feature Propagation for Referring Segmentation](https://mlsanity.com/paper/2204.12109/) (2022)
- [TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer](https://mlsanity.com/paper/2206.06619/) (2022)
- [Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation](https://mlsanity.com/paper/2305.15302/) (2023)
- [Parallel Vertex Diffusion for Unified Visual Grounding](https://mlsanity.com/paper/2303.07216/) (2023)
- [Connecting What to Say With Where to Look by Modeling Human Attention Traces](https://mlsanity.com/paper/2105.05964/) (2021)
- [Correspondence Matters for Video Referring Expression Comprehension](https://mlsanity.com/paper/2207.10400/) (2022)
- [Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching](https://mlsanity.com/paper/2201.06686/) (2022)
- [A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension](https://mlsanity.com/paper/2204.07913/) (2022)
- [LRVS-Fashion: Extending Visual Search with Referring Instructions](https://mlsanity.com/paper/2306.02928/) (2023)
- [Optimizing Multispectral Object Detection: A Bag of Tricks and Comprehensive Benchmarks](https://mlsanity.com/paper/2411.18288/) (2024)
- [1st Place Solution for 5th LSVOS Challenge: Referring Video Object Segmentation](https://mlsanity.com/paper/2401.00663/) (2024)
- [From Coarse to Fine-grained Concept based Discrimination for Phrase Detection](https://mlsanity.com/paper/2112.03237/) (2021)

## Abstract (arXiv)

Associating image regions with text queries has been recently explored as a new way to bridge visual and linguistic representations. A few pioneering approaches have been proposed based on recurrent neural language models trained generatively (e.g., generating captions), but achieving somewhat limited localization accuracy. To better address natural-language-based visual entity localization, we propose a discriminative approach. We formulate a discriminative bimodal neural network (DBNet), which can be trained by a classifier with extensive use of negative samples. Our training objective encourages better localization on single images, incorporates text phrases in a broad range, and properly pairs image regions with text phrases into positive and negative examples. Experiments on the Visual Genome dataset demonstrate the proposed DBNet significantly outperforms previous state-of-the-art methods both for localization on single images and for detection on multiple images. We we also establish an evaluation protocol for natural-language visual detection.