mlsanityData to 14 Oct 2026

Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning

arXiv 2503.18013 ↗First posted 23 Mar 2025cs.CV cs.AIcited by 69

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) typically follow a two-stage training paradigm-pretraining and supervised fine-tuning. Recently, preference optimization, derived from the language domain, has emerged as an effective post-training reinforcement strategy to enhance capabilities of LVLMs. However, constructing high-quality human-annotated preference data and developing robust reward models to mimic these preferences are both costly and challenging. Motivated by this observation, we propose Vision-R1, a novel vision-guided R1-like reinforcement learning algorithm for LVLMs that rewards models with definitive vision feedback. It only leverages curated instruction data, eliminating the need for specialized reward models and handcrafted preference datasets. We incorporate a criterion-driven reward function that further integrates multi-dimensional feedback to evaluate model completions comprehensively based on the vision task logic. Furthermore, we introduce a progressive rule refinement strategy that dynamically adjusts the reward criteria during training, enabling continuous model improvement and mitigating reward hacking. Extensive experiments on both in-distribution and out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with Vision-R1 achieves consistent performance gains, with even up to 50% improvement and surpassing the state-of-the-art 10x size model.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 53 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

45 of 53 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  12. 2025Qwen2.5-VL Technical Reportcited by 5,587
33 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  13. 2021Grounded Language-Image Pre-trainingcited by 1,156
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  15. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  16. 2016A Diagram Is Worth A Dozen Imagescited by 804
  17. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  18. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  19. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  20. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  21. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  22. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  23. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  24. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  25. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  26. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  27. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  28. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  29. 2023GREC: Generalized Referring Expression Comprehensioncited by 42
  30. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Modelscited by 31
  31. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  32. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
  33. 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 46; 2026: 23.

69 papers.

Sort:

By area:

  1. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  3. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  4. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  5. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 52
  6. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  7. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  8. 2025Reinforced Visual Perception with Toolscited by 38
  9. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 38
  10. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  11. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  12. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
38 more
  1. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  2. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 26
  3. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thoughtcited by 24
  4. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  5. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 21
  6. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  7. 2025CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Rewardcited by 15
  8. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  9. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agentscited by 15
  10. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  11. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 14
  12. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  13. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  14. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curationcited by 13
  15. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  16. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  17. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
  18. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  19. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  20. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 7
  21. 2025Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraintscited by 6
  22. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  23. 2025PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Designcited by 6
  24. 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
  25. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Modelscited by 6
  26. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learningcited by 5
  27. 2025BTL-UI: Blink-Think-Link Reasoning Model for GUI Agentcited by 5
  28. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  29. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  30. 2026Towards One-to-Many Temporal Groundingcited by 4
  31. 2025Align and Surpass Human Camouflaged Perception: Visual Refocus Reinforcement Fine-Tuningcited by 3
  32. 2025Unleashing Perception-Time Scaling to Multimodal Reasoning Modelscited by 3
  33. 2025Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Groundingcited by 3
  34. 2025REVEALER: Reinforcement-Guided Visual Reasoning for Element-Level Text-Image Alignment Evaluationcited by 3
  35. 2026HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Frameworkcited by 3
  36. 2026Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Viewscited by 3
  37. 2025Empowering Lightweight MLLMs with Reasoning via Long CoT SFTcited by 2
  38. 2026Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video Reasoningcited by 2

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  3. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
  4. 2025SPARK: Synergistic Policy And Reward Co-Evolving Framework
  5. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  6. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
  7. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning
  8. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences