Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding
Not on the map. Why?
Abstract (arXiv)
Visual grounding aims to localize the image regions based on a textual query. Given the difficulty of large-scale data curation, we investigate how to effectively learn visual grounding under data-scarce settings in this paper. To address the data scarcity, we propose a novel framework, POBF (Paint Outside the Box and Filter). POBF synthesizes images by inpainting outside the box, tackling a label misalignment issue encountered in previous works. Furthermore, POBF leverages an innovative filtering scheme to select the most effective training data. This scheme combines a hardness score and an overfitting score, balanced by a penalty term. Extensive experiments across four benchmark datasets demonstrate that POBF consistently improves performance, achieving an average gain of 5.83% over the real-data-only method and outperforming leading baselines by 2.29%-3.85% in accuracy. Additionally, we validate the robustness and generalizability of POBF across various generative models, training data sizes, and model architectures.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 99 of its 123 references that are in the map. First posted on arXiv 1 Dec 2024.
Builds on
99 of 123 references.
References · 99
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023Segment Anythingcited by 8,158
- 2023Visual Instruction Tuningcited by 7,792
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
- 2016Modeling Context in Referring Expressionscited by 1,054
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
38 more
- 2017A Closer Look at Memorization in Deep Networkscited by 936
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
- 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
- 2021Deduplicating Training Data Makes Language Models Bettercited by 705
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
- 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
- 2018Recognition in Terra Incognitacited by 655
- 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
- 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
- 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
- 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
- 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
- 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
- 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
- 2023Data Selection for Language Models via Importance Resamplingcited by 322
- 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
- 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 292
- 2023Effective Data Augmentation With Diffusion Modelscited by 282
- 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
- 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
- 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
- 2017Decoupling "when to update" from "how to update"cited by 230
- 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
- 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
- 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
- 2016Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?cited by 194
- 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
- 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
- 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
- 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
- 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
- 2023Open-vocabulary Object Segmentation with Diffusion Modelscited by 82
- 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
- 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
- 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
50 most cited of 99 All 99 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.