mlsanityData to 7 Oct 2026

SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning

arXiv 2506.01713 ↗First posted 2 Jun 2025cs.CLcited by 53

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful and instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose Multimodal Self-Reflection enhanced reasoning with Group Relative Policy Optimization (SRPO), a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks, including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 80 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

56 of 80 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  3. 2025Kimi-VL Technical Report10 Apr
  4. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models26 Mar
  5. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles21 Mar
  6. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  7. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
  8. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  9. 2025Virgo: A Preliminary Exploration on Reproducing o1-like MLLM3 Jan
  10. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  11. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  12. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
3 more
  1. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 41

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2024OpenAI o1 System Cardcited by 1,220
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  12. 2024DeepSeek-V3 Technical Reportcited by 724
29 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  4. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  5. 2025Seed1.5-VL Technical Reportcited by 367
  6. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  7. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
  8. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  9. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  10. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  11. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  12. 2023Efficient Large Language Models: A Surveycited by 183
  13. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  14. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  15. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  16. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkcited by 70
  17. 2025Rethinking Reflection in Pre-Trainingcited by 43
  18. 2024Autoregressive Models in Vision: A Surveycited by 39
  19. 2025PhyX: Does Your Model Have the "Wits" for Physical Reasoning?cited by 37
  20. 2025Efficient Diffusion Models: A Surveycited by 35
  21. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
  22. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  23. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  24. 2025Entropy-based Exploration Conduction for Multi-step Reasoningcited by 28
  25. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  26. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
  27. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  28. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  29. 2025Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 25.

53 papers.

Sort:

By area:

  1. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  2. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  3. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversitycited by 41
  4. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 37
  5. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  7. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  8. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  9. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  10. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 17
  11. 2025REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understandingcited by 15
  12. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
38 more
  1. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Searchcited by 11
  2. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  3. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 11
  4. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  5. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  6. 2026Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoningcited by 9
  7. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  8. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 8
  9. 2025VIDEOP2R: Video Understanding from Perception to Reasoningcited by 8
  10. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  11. 2025Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoningcited by 7
  12. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 7
  13. 2026Towards Explainable Industrial Anomaly Detection via Knowledge-Guided Latent Reasoningcited by 5
  14. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5
  15. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4
  16. 2025Boosting Reasoning in Large Multimodal Models via Activation Replaycited by 4
  17. 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshapingcited by 4
  18. 2025VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Groundingcited by 3
  19. 2025DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuningcited by 3
  20. 2026Human Simulation Computation: A Human-Inspired Framework for Adaptive AI Systemscited by 3
  21. 2026What does RL improve for Visual Reasoning? A Frankenstein-Style Analysiscited by 3
  22. 2026OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Modelcited by 3
  23. 2026Learning Self-Correction in Vision-Language Models via Rollout Augmentationcited by 2
  24. 2025Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimizationcited by 1
  25. 2025MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlookcited by 1
  26. 2025Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Modelscited by 1
  27. 2025Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Modelscited by 1
  28. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  29. 2026M3-AD: Reflection-aware Multi-modal, Multi-category, and Multi-dimensional Benchmark and Framework for Industrial Anomaly Detectioncited by 1
  30. 2026One Model, Two Minds: Task-Conditioned Reasoning for Unified Image Quality and Aesthetic Assessmentcited by 1
  31. 2026AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolutioncited by 1
  32. 2026I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoningcited by 1
  33. 2025EvoMail: Self-Evolving Cognitive Agents for Adaptive Spam and Phishing Email Defensecited by 0
  34. 2026SRVAU-R1: Enhancing Video Anomaly Understanding via Reflection-Aware Learningcited by 0
  35. 2026From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Modelscited by 0
  36. 2026Reflection Anchors for Propagation-Aware Visual Retention in Long-Chain Multimodal Reasoningcited by 0
  37. 2026Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluationcited by 0
  38. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start
  2. 2026GRPO and Reflection Reward for Mathematical Reasoning in Large Language Models
  3. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
  5. 2025MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization
  6. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
  7. 2025OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning
  8. 2025Reinforce LLM Reasoning through Multi-Agent Reflection