mlsanityData to 7 Oct 2026

R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization

arXiv 2503.12937 ↗First posted 17 Mar 2025cs.AI cs.CL cs.CVcited by 296

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning paths are. In this work, we aim to enhance the MLLMs' reasoning ability beyond passively imitating positive reasoning paths. To this end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new online reinforcement learning framework that enables MLLMs to self-improve reasoning ability via simple, effective and dense step-wise rewarding. Specifically, StepGRPO introduces two novel rule-based reasoning rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary intermediate reasoning steps via a soft key-step matching technique, while StepRAR rewards reasoning paths that follow a well-structured and logically consistent reasoning process through a reasoning completeness and logic evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive experiments over 8 benchmarks demonstrate the superiority of our methods.

Key facts. Cited by 296 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 57 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

47 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
30 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  12. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  13. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  14. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  15. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  16. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  17. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  18. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  19. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  20. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  21. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  22. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  23. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  24. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  25. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  26. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  27. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  28. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  29. 2025GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Imagescited by 35
  30. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 183; 2026: 113.

296 papers.

Sort:

By area:

  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  2. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025Agentic Reinforced Policy Optimizationcited by 166
  5. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  7. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  8. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  9. 2025Latent Visual Reasoningcited by 99
  10. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  11. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 94
  12. 2025Q-Insight: Understanding Image Quality via Visual Reinforcement Learningcited by 93
38 more
  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  2. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  3. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  4. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 86
  5. 2025Visual Planning: Let's Think Only with Imagescited by 85
  6. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  7. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  8. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  9. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  10. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 65
  11. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  12. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  13. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  14. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  15. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  16. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  17. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 51
  18. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  19. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  20. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  21. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  22. 2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
  23. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  24. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  25. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  26. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 42
  27. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 37
  28. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  29. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  30. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  31. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  32. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  33. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
  34. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  35. 2025ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RLcited by 30
  36. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  37. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  38. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28

50 most cited of 296 All 296 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  3. 2025SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  5. 2026LamPO: A Lambda Style Policy Optimization for Reasoning Language Models
  6. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
  7. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  8. 2025Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning