mlsanityData to 7 Oct 2026

Learning from Synthetic Data for Visual Grounding

arXiv 2403.13804 ↗First posted 20 Mar 2024cs.CV cs.CL cs.LGcited by 4

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

This paper extensively investigates the effectiveness of synthetic training data to improve the capabilities of vision-and-language models for grounding textual descriptions to image regions. We explore various strategies to best generate image-text pairs and image-text-box triplets using a series of pretrained models under different settings and varying degrees of reliance on real data. Through comparative analyses with synthetic, real, and web-crawled data, we identify factors that contribute to performance differences, and propose SynGround, an effective pipeline for generating useful synthetic data for visual grounding. Our findings show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models and offers the potential for arbitrarily large scale data generation. Particularly, data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively, across the RefCOCO+ and the Flickr30k benchmarks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 59 of its 71 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

59 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models30 Jan
  2. 2021Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data30 Nov

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2016Playing for Data: Ground Truth from Computer Gamescited by 816
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2017VisDA: The Visual Domain Adaptation Challengecited by 432
  13. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  14. 2017Learning from Synthetic Humanscited by 358
  15. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  16. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  17. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  18. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  19. 2023Teaching CLIP to Count to Tencited by 194
  20. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  21. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  22. 2017Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenescited by 122
  23. 2018Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approachcited by 119
  24. 2021LENS: Localization enhanced by NeRF synthesiscited by 116
  25. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  26. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  27. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  28. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  29. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  30. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  31. 2019Speech Recognition with Augmented Synthesized Speechcited by 60
  32. 2019Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignmentcited by 51
  33. 2019Generative adversarial networks (GAN) based efficient sampling of chemical space for inverse design of inorganic materialscited by 46
  34. 2019Synthetic Humans for Action Recognition from Unseen Viewpointscited by 46
  35. 2019Generating Synthetic Audio Data for Attention-Based Speech Recognition Systemscited by 46
  36. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  37. 2022Pseudo-Q: Generating Pseudo Language Queries for Visual Groundingcited by 33
  38. 2018Multi-level Multimodal Common Semantic Space for Image-Phrase Groundingcited by 32

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Datacited by 4
  2. 2026Grounding Synthetic Data Generation With Vision and Language Modelscited by 3
  3. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  4. 2025Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Groundingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
  2. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  3. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  4. 2025Zero-Shot 3D Visual Grounding from Vision-Language Models
  5. 2025Towards Understanding Visual Grounding in Visual Language Models
  6. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding
  7. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
  8. 2024Learning Visual Grounding from Generative Vision and Language Model