# Zero-Shot Grounding of Objects from Natural Language Queries

arXiv: https://arxiv.org/abs/1908.07129 · HTML: https://mlsanity.com/paper/1908.07129/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2019.

## Built on by (most cited first, 30 of 79)

- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain](https://mlsanity.com/paper/2401.16822/) (2024)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [SAT: 2D Semantics Assisted Training for 3D Visual Grounding](https://mlsanity.com/paper/2105.11450/) (2021)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension](https://mlsanity.com/paper/2204.05991/) (2022)
- [Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation](https://mlsanity.com/paper/2105.01839/) (2021)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)
- [Vision-Language Models in Remote Sensing: Current Progress and Future Trends](https://mlsanity.com/paper/2305.05726/) (2023)
- [Open-vocabulary Object Segmentation with Diffusion Models](https://mlsanity.com/paper/2301.05221/) (2023)
- [Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning](https://mlsanity.com/paper/2205.00272/) (2022)
- [Contrastive Grouping with Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2309.01017/) (2023)
- [Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding](https://mlsanity.com/paper/2104.04386/) (2021)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Visual Grounding with Transformers](https://mlsanity.com/paper/2105.04281/) (2021)
- [TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer](https://mlsanity.com/paper/2206.06619/) (2022)
- [Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding](https://mlsanity.com/paper/2211.14241/) (2022)
- [Described Object Detection: Liberating Object Detection with Flexible Expressions](https://mlsanity.com/paper/2307.12813/) (2023)
- [RemoteSAM: Towards Segment Anything for Earth Observation](https://mlsanity.com/paper/2505.18022/) (2025)
- [3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2308.16632/) (2023)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [Context Disentangling and Prototype Inheriting for Robust Visual Grounding](https://mlsanity.com/paper/2312.11967/) (2023)
- [Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter](https://mlsanity.com/paper/2311.05779/) (2023)
- [Mono3DVG: 3D Visual Grounding in Monocular Images](https://mlsanity.com/paper/2312.08022/) (2023)

## Abstract (arXiv)

A phrase grounding system localizes a particular object in an image referred to by a natural language query. In previous work, the phrases were restricted to have nouns that were encountered in training, we extend the task to Zero-Shot Grounding(ZSG) which can include novel, "unseen" nouns. Current phrase grounding systems use an explicit object detection network in a 2-stage framework where one stage generates sparse proposals and the other stage evaluates them. In the ZSG setting, generating appropriate proposals itself becomes an obstacle as the proposal generator is trained on the entities common in the detection and grounding datasets. We propose a new single-stage model called ZSGNet which combines the detector network and the grounding system and predicts classification scores and regression parameters. Evaluation of ZSG system brings additional subtleties due to the influence of the relationship between the query and learned categories; we define four distinct conditions that incorporate different levels of difficulty. We also introduce new datasets, sub-sampled from Flickr30k Entities and Visual Genome, that enable evaluations for the four conditions. Our experiments show that ZSGNet achieves state-of-the-art performance on Flickr30k and ReferIt under the usual "seen" settings and performs significantly better than baseline in the zero-shot setting.