# One RL to See Them All: Visual Triple Unified Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.18129 · HTML: https://mlsanity.com/paper/2505.18129/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 37)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025)
- [VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning](https://mlsanity.com/paper/2504.08837/) (2025)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs](https://mlsanity.com/paper/2406.18521/) (2024)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning](https://mlsanity.com/paper/2501.00321/) (2024)
- [Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models](https://mlsanity.com/paper/2408.15556/) (2024)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [Perception-R1: Pioneering Perception Policy with Reinforcement Learning](https://mlsanity.com/paper/2504.07954/) (2025)
- [ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering](https://mlsanity.com/paper/2504.05506/) (2025)
- [On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering](https://mlsanity.com/paper/2002.10215/) (2020)
- [TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains](https://mlsanity.com/paper/2404.19205/) (2024)
- [PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns](https://mlsanity.com/paper/2403.13315/) (2024)
- [VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge](https://mlsanity.com/paper/2504.10342/) (2025)

## Built on by (most cited first, 30 of 30)

- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [Perception-Aware Policy Optimization for Multimodal Reasoning](https://mlsanity.com/paper/2507.06448/) (2025)
- [Look-Back: Implicit Visual Re-focusing in MLLM Reasoning](https://mlsanity.com/paper/2507.03019/) (2025)
- [Spotlight on Token Perception for Multimodal Reinforcement Learning](https://mlsanity.com/paper/2510.09285/) (2025)
- [MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.21475/) (2025)
- [LongCat-Next: Lexicalizing Modalities as Discrete Tokens](https://mlsanity.com/paper/2603.27538/) (2026)
- [Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning](https://mlsanity.com/paper/2510.12712/) (2025)
- [Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle](https://mlsanity.com/paper/2508.05612/) (2025)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought](https://mlsanity.com/paper/2603.22847/) (2026)
- [Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning](https://mlsanity.com/paper/2509.22746/) (2025)
- [Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models](https://mlsanity.com/paper/2604.01840/) (2026)
- [Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start](https://mlsanity.com/paper/2510.25801/) (2025)
- [Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimization](https://mlsanity.com/paper/2602.03380/) (2026)
- [Large Vision-Language Models Get Lost in Attention](https://mlsanity.com/paper/2605.05668/) (2026)
- [PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping](https://mlsanity.com/paper/2606.08708/) (2026)
- [Visual Programmability: A Guide for Code-as-Thought in Chart Understanding](https://mlsanity.com/paper/2509.09286/) (2025)
- [EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence](https://mlsanity.com/paper/2510.20578/) (2025)
- [Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization](https://mlsanity.com/paper/2602.21743/) (2026)
- [Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization](https://mlsanity.com/paper/2605.27906/) (2026)
- [PairUni: Pairwise Training for Unified Multimodal Language Models](https://mlsanity.com/paper/2510.25682/) (2025)
- [Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding](https://mlsanity.com/paper/2609.09300/) (2026)
- [Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift](https://mlsanity.com/paper/2609.39183/) (2026)
- [Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View](https://mlsanity.com/paper/2511.06722/) (2025)
- [Weather-R1: Logically Consistent Reinforcement Fine-Tuning for Multimodal Reasoning in Meteorology](https://mlsanity.com/paper/2601.14044/) (2026)
- [ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoning](https://mlsanity.com/paper/2602.02004/) (2026)
- [CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models](https://mlsanity.com/paper/2603.21077/) (2026)
- [MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Grounding](https://mlsanity.com/paper/2603.28120/) (2026)
- [EviRover: Reinforcing Agentic Perception Beyond a Glance](https://mlsanity.com/paper/2609.40230/) (2026)
- [GeoPID: Decomposing and Steering Visual Information in Vision-Language Models](https://mlsanity.com/paper/2610.08401/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is becoming an important direction for post-training vision-language models (VLMs), but public training methodologies for unified multimodal RL remain much less mature, especially for heterogeneous reasoning and perception-heavy tasks. We propose V-Triune, a Visual Triple Unified Reinforcement Learning methodology for unified multimodal RL. It organizes training around three coordinated abstractions: Sample-Level Reward Routing, Verifier-Level Outcome Verification, and Source-Level Diagnostics. Within this methodology, Dynamic IoU provides localization-specific reward shaping that avoids reward ambiguity under loose thresholds and reward sparsity under strict ones. Built on V-Triune, we develop Orsta (7B, 32B), a family of models jointly trained on eight reasoning and perception tasks. Under matched budgets, unified training matches or outperforms specialist mixtures. The final Orsta models improve over their backbones on MEGA-Bench, compare favorably with strong multi-task RL-VLM baselines, and transfer these gains to a broad set of downstream benchmarks. These results show that unified RL can improve both reasoning and perception within a single VLM RL pipeline.The V-Triune system, along with the Orsta models, is publicly available at https://github.com/MiniMax-AI/One-RL-to-See-Them-All.