mlsanityData to 7 Oct 2026

VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders

arXiv 2309.01141 ↗First posted 3 Sep 2023cs.CVcited by 18

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Large-scale text-to-image diffusion models have shown impressive capabilities for generative tasks by leveraging strong vision-language alignment from pre-training. However, most vision-language discriminative tasks require extensive fine-tuning on carefully-labeled datasets to acquire such alignment, with great cost in time and computing resources. In this work, we explore directly applying a pre-trained generative diffusion model to the challenging discriminative task of visual grounding without any fine-tuning and additional training dataset. Specifically, we propose VGDiffZero, a simple yet effective zero-shot visual grounding framework based on text-to-image diffusion models. We also design a comprehensive region-scoring method considering both global and local contexts of each isolated proposal. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that VGDiffZero achieves strong performance on zero-shot visual grounding. Our code is available at https://github.com/xuyang-liu16/VGDiffZero.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 22 of its 25 references that are in the map. First posted on arXiv 3 Sep 2023.

Builds on

22 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners18 May
  2. 2023Your Diffusion Model is Secretly a Zero-Shot Classifier28 Mar
  3. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2016Modeling Context in Referring Expressionscited by 1,054
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
6 more
  1. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  2. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  3. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 63
  4. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  5. 2023Language Adaptive Weight Generation for Multi-task Visual Groundingcited by 35
  6. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 4; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  2. 2023UniGS: Unified Representation for Image Generation and Segmentationcited by 29
  3. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  4. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  5. 2024SATO: Stable Text-to-Motion Frameworkcited by 20
  6. 2024Exploring Phrase-Level Grounding with Text-to-Image Diffusion Modelcited by 13
  7. 2024M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehensioncited by 11
  8. 2024Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Modelscited by 7
  9. 2024MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehensioncited by 6
  10. 2024DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Groundingcited by 5
  11. 2023DAP: Domain-aware Prompt Learning for Vision-and-Language Navigationcited by 4
  12. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentationcited by 4
6 more
  1. 2024Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniquescited by 2
  2. 2025GS: Generative Segmentation via Label Diffusioncited by 2
  3. 2025In-Situ Tweedie Discrete Diffusion Modelscited by 2
  4. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoningcited by 1
  5. 2026Prompting Diffusion Models for Zero-Shot Instance Segmentationcited by 0
  6. 2026Attention Dynamics in Diffusion Models: A Visual Analytics Framework for Human-AI Collaborationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Language-Guided Diffusion Model for Visual Grounding
  2. 2023GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection
  3. 2023Advancing Visual Grounding with Scene Knowledge: Benchmark and Method
  4. 2025Towards Understanding Visual Grounding in Visual Language Models
  5. 2023Text-to-Image Diffusion Models are Zero-Shot Classifiers
  6. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  7. 2025Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation
  8. 2026High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding