# Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning

arXiv: https://arxiv.org/abs/2607.02490 · HTML: https://mlsanity.com/paper/2607.02490/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 30)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models](https://mlsanity.com/paper/2406.09403/) (2024)
- [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025)
- [ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering](https://mlsanity.com/paper/2504.05506/) (2025)
- [VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use](https://mlsanity.com/paper/2505.19255/) (2025)
- [Visual Planning: Let's Think Only with Images](https://mlsanity.com/paper/2505.11409/) (2025)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)
- [SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning](https://mlsanity.com/paper/2506.01713/) (2025)
- [Look-Back: Implicit Visual Re-focusing in MLLM Reasoning](https://mlsanity.com/paper/2507.03019/) (2025)
- [VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs](https://mlsanity.com/paper/2407.01863/) (2024)
- [Learning to Refine with Fine-Grained Natural Language Feedback](https://mlsanity.com/paper/2407.02397/) (2024)
- [v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning](https://mlsanity.com/paper/2505.18842/) (2025)
- [GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior](https://mlsanity.com/paper/2506.08012/) (2025)
- [Bridge the Modality and Capability Gaps in Vision-Language Model Selection](https://mlsanity.com/paper/2403.13797/) (2024)
- [REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understanding](https://mlsanity.com/paper/2511.13026/) (2025)
- [VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning](https://mlsanity.com/paper/2412.02172/) (2024)
- [TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning](https://mlsanity.com/paper/2506.08646/) (2025)
- [Thinking with Images via Self-Calling Agent](https://mlsanity.com/paper/2512.08511/) (2025)
- [Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine](https://mlsanity.com/paper/2603.06665/) (2026)
- [SkillFactory: Self-Distillation For Learning Cognitive Behaviors](https://mlsanity.com/paper/2512.04072/) (2025)
- [Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models](https://mlsanity.com/paper/2603.27201/) (2026)
- [Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework](https://mlsanity.com/paper/2603.07659/) (2026)

## Abstract (arXiv)

Large vision-language models can reason over multimodal inputs by generating textual chains of thought (CoT). A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors. However, existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections, especially for out-of-distribution images. To address this issue, we propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection. First, we randomly mask trajectory prefixes during training to emphasize recovery from incorrect intermediate predictions rather than making early mistakes. Second, we introduce buffered roll-ins from an experience replay buffer to expose the model to diverse failure states that it must learn to correct. We evaluate our approach on visual grounding tasks involving tables and charts, as well as spatial navigation benchmarks. While off-the-shelf and conventionally fine-tuned models degrade substantially under distribution shift, our method substantially improves average out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines by using self-reflection effectively.