Data to 7 Oct 2026

SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

arXiv 2504.11468 ↗First posted 10 Apr 2025cs.CLcited by 209

On the map: LLM reasoning → Multimodal reasoning

also in 1 more

Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent RL by inducing ``pseudo reasoning paths'' imitated from expert models. While these paths may resemble the native reasoning paths of RL models, they often involve prolonged, hesitant, less informative steps, and incorrect reasoning. To systematically study this effect, we introduce VLAA-Thinking, a new multimodal dataset designed to support reasoning in LVLMs. Constructed via a six-step pipeline involving captioning, reasoning distillation, answer rewrite and verification, VLAA-Thinking comprises high-quality, step-by-step visual reasoning traces for SFT, along with a more challenging RL split from the same data source. Using this dataset, we conduct extensive experiments comparing SFT, RL and their combinations. Results show that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, building on the Group Relative Policy Optimization (GRPO) with a novel mixed reward module integrating both perception and cognition signals, our RL approach fosters more genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard) among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope our findings provide valuable insights in developing reasoning-capable LVLMs and can inform future research in this area.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 58 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

48 of 58 references.

Closest ancestors · 18

Same direction, newest first

6 more

Other references · 30

2020Language Models are Few-Shot Learnerscited by 32,199
2017Proximal Policy Optimization Algorithmscited by 14,441
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655

18 more

2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
2023Reward Design with Language Modelscited by 257
2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
2025Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 136; 2026: 73.

209 papers.

Sort:

By area:

2025Learning to Reason under Off-Policy Guidancecited by 266
2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
2025GRIT: Teaching MLLMs to Think with Imagescited by 121
2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
2025Latent Visual Reasoningcited by 99
2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67

38 more

2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 51
2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
2025Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Rewardcited by 45
2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
2025Reinforced Visual Perception with Toolscited by 38
2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 38
2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
2025Time Series Forecasting via Reasoning: A Slow-Thinking Approach with Reinforcement Fine-Tuned LLMscited by 34
2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 33
2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 32
2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoningcited by 29
2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26
2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 26
2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillationcited by 26
2025V-Thinker: Interactive Thinking with Imagescited by 26
2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
2025Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Spacecited by 25
2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 24
2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
2025WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learningcited by 21
2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
2025Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoningcited by 20
2025Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shufflecited by 19
2025A Survey on Agentic Multimodal Large Language Modelscited by 19
2026VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twicecited by 19

50 most cited of 209 All 209 (JSON)

Similar, not cited

Close in wording, no citation either way