# Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?

arXiv: https://arxiv.org/abs/2404.06510 · HTML: https://mlsanity.com/paper/2404.06510/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 46 of its 74 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)

## Built on by (most cited first, 20 of 20)

- [VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use](https://mlsanity.com/paper/2505.19255/) (2025)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception](https://mlsanity.com/paper/2504.15362/) (2025)
- [Perception in Reflection](https://mlsanity.com/paper/2504.07165/) (2025)
- [DatBench: Discriminative, Faithful, and Efficient VLM Evaluations](https://mlsanity.com/paper/2601.02316/) (2026)
- [Pricing and Competition for Generative AI](https://mlsanity.com/paper/2411.02661/) (2024)
- [FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models](https://mlsanity.com/paper/2407.11522/) (2024)
- [Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions](https://mlsanity.com/paper/2506.08927/) (2025)
- [FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization](https://mlsanity.com/paper/2605.31145/) (2026)
- [Guideline-Consistent Segmentation via Multi-Agent Refinement](https://mlsanity.com/paper/2509.04687/) (2025)
- [Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale](https://mlsanity.com/paper/2511.05705/) (2025)
- [Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQA](https://mlsanity.com/paper/2605.10850/) (2026)
- [Iterative Visual Thinking and the Self-Correction Mirage in VLM Grounding](https://mlsanity.com/paper/2606.13156/) (2026)
- [SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning](https://mlsanity.com/paper/2607.10966/) (2026)
- [Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition](https://mlsanity.com/paper/2608.23869/) (2026)
- [FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs](https://mlsanity.com/paper/2606.04282/) (2026)
- [VisCritic: Visual State Comparison as Process Reward for GUI Agents](https://mlsanity.com/paper/2606.24525/) (2026)
- [InsightSeg: Reusing Correction Insights for Guideline-Consistent Segmentation](https://mlsanity.com/paper/2609.02002/) (2026)
- [Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding](https://mlsanity.com/paper/2609.04741/) (2026)
- [CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement](https://mlsanity.com/paper/2610.01710/) (2026)

## Abstract (arXiv)

Enhancing semantic grounding abilities in Vision-Language Models (VLMs) often involves collecting domain-specific training data, refining the network architectures, or modifying the training recipes. In this work, we venture into an orthogonal direction and explore whether VLMs can improve their semantic grounding by "receiving" feedback, without requiring in-domain data, fine-tuning, or modifications to the network architectures. We systematically analyze this hypothesis using a feedback mechanism composed of a binary signal. We find that if prompted appropriately, VLMs can utilize feedback both in a single step and iteratively, showcasing the potential of feedback as an alternative technique to improve grounding in internet-scale VLMs. Furthermore, VLMs, like LLMs, struggle to self-correct errors out-of-the-box. However, we find that this issue can be mitigated via a binary verification mechanism. Finally, we explore the potential and limitations of amalgamating these findings and applying them iteratively to automatically enhance VLMs' grounding performance, showing grounding accuracy consistently improves using automated feedback across all models in all settings investigated. Overall, our iterative framework improves semantic grounding in VLMs by more than 15 accuracy points under noise-free feedback and up to 5 accuracy points under a simple automated binary verification mechanism. The project website is hosted at https://andrewliao11.github.io/vlms_feedback