mlsanityData to 14 Oct 2026

VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

arXiv 2504.08837 ↗First posted 10 Apr 2025cs.LG cs.AIcited by 323

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated great potential in solving challenging problems through explicit reflection. They significantly outperform the best fast-thinking models, such as GPT-4o, on various math and science benchmarks. However, their multimodal reasoning capabilities remain on par with fast-thinking models. For instance, GPT-o1's performance on benchmarks like MathVista, MathVerse, and MathVision is similar to fast-thinking models. In this paper, we aim to enhance the slow-thinking capabilities of vision-language models using reinforcement learning (without relying on distillation) to advance the state of the art. First, we adapt the GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to address the vanishing advantages problem. While this approach yields strong performance, the resulting RL-trained models exhibit limited self-reflection or self-verification. To further encourage slow-thinking, we introduce Forced Rethinking, which appends a rethinking trigger token to the end of rollouts in RL training, explicitly enforcing a self-reflection reasoning step. By combining these two techniques, our model, VL-Rethinker, advances state-of-the-art scores on MathVista, MathVerse to achieve 80.4%, 63.5% respectively. VL-Rethinker also achieves open-source SoTA on multi-disciplinary benchmarks such as MathVision, MMMU-Pro, EMMA, and MEGA-Bench, narrowing the gap with OpenAI-o1. Our empirical results show the effectiveness of our approaches.

Key facts. Cited by 323 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 63 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

40 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  2. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
  3. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  4. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark9 Jan
  5. 2025Virgo: A Preliminary Exploration on Reproducing o1-like MLLM3 Jan
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  7. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale6 Dec
  8. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
19 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2025s1: Simple test-time scalingcited by 1,341
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2016A Diagram Is Worth A Dozen Imagescited by 804
  10. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  13. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  14. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  15. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  16. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  17. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Taskscited by 41
  18. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 35
  19. 2022ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 159; 2026: 164.

323 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  3. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  4. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  5. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  6. 2025Latent Visual Reasoningcited by 104
  7. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
  8. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  9. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  10. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 87
  11. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  12. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
38 more
  1. 2025VGR: Visual Grounded Reasoningcited by 73
  2. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  3. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  4. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  5. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  6. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  7. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  8. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  9. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 53
  10. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  11. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 49
  12. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  13. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 43
  14. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  15. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  16. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversitycited by 41
  17. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 40
  18. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 39
  19. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
  20. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  21. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  22. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 36
  23. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  24. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  25. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  26. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  27. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 33
  28. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  29. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domainscited by 31
  30. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 30
  31. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 30
  32. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  33. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  34. 2025Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Spacecited by 28
  35. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 27
  36. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 27
  37. 2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
  38. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25

50 most cited of 323 All 323 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models
  2. 2025Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning
  3. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
  4. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  5. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
  6. 2025Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
  7. 2026Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
  8. 2026Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning