# Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency

arXiv: https://arxiv.org/abs/2506.01908 · HTML: https://mlsanity.com/paper/2506.01908/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 41 of its 41 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 41)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization](https://mlsanity.com/paper/2503.12937/) (2025)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [VTimeLLM: Empower LLM to Grasp Video Moments](https://mlsanity.com/paper/2311.18445/) (2023)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model](https://mlsanity.com/paper/2503.05132/) (2025)
- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [Can I Trust Your Answer? Visually Grounded Video Question Answering](https://mlsanity.com/paper/2309.01327/) (2023)
- [Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input](https://mlsanity.com/paper/2408.15542/) (2024)
- [ST-LLM: Large Language Models Are Effective Temporal Learners](https://mlsanity.com/paper/2404.00308/) (2024)

## Built on by (most cited first, 26 of 26)

- [Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning](https://mlsanity.com/paper/2508.04416/) (2025)
- [OneThinker: All-in-one Reasoning Model for Image and Video](https://mlsanity.com/paper/2512.03043/) (2025)
- [Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning](https://mlsanity.com/paper/2510.23473/) (2025)
- [Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools](https://mlsanity.com/paper/2510.08480/) (2025)
- [EditThinker: Unlocking Iterative Reasoning for Any Image Editor](https://mlsanity.com/paper/2512.05965/) (2025)
- [AdaTooler-V: Adaptive Tool-Use for Images and Videos](https://mlsanity.com/paper/2512.16918/) (2025)
- [VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice](https://mlsanity.com/paper/2601.05175/) (2026)
- [SpaceVista: All-Scale Visual Spatial Reasoning from mm to km](https://mlsanity.com/paper/2510.09606/) (2025)
- [AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video](https://mlsanity.com/paper/2508.03100/) (2025)
- [VideoNSA: Native Sparse Attention Scales Video Understanding](https://mlsanity.com/paper/2510.02295/) (2025)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models](https://mlsanity.com/paper/2510.05034/) (2025)
- [ViSS-R1: Self-Supervised Reinforcement Video Reasoning](https://mlsanity.com/paper/2511.13054/) (2025)
- [TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding](https://mlsanity.com/paper/2508.07683/) (2025)
- [TimeScope: Towards Task-Oriented Temporal Grounding In Long Videos](https://mlsanity.com/paper/2509.26360/) (2025)
- [GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking](https://mlsanity.com/paper/2602.17555/) (2026)
- [Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning](https://mlsanity.com/paper/2606.27828/) (2026)
- [TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2607.17423/) (2026)
- [DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding](https://mlsanity.com/paper/2605.26680/) (2026)
- [Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs](https://mlsanity.com/paper/2606.18441/) (2026)
- [CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs](https://mlsanity.com/paper/2606.19927/) (2026)
- [SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards](https://mlsanity.com/paper/2606.24726/) (2026)
- [Multi-Agent Self-Improving Reinforcement Learning for Video Reasoning](https://mlsanity.com/paper/2608.28675/) (2026)
- [Controllable Multi-label Video Safety Detection via Adaptive Tversky Policy Optimization](https://mlsanity.com/paper/2610.02019/) (2026)
- [VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Research](https://mlsanity.com/paper/2610.04911/) (2026)
- [Less from More: Reinforcing Sparse Video Reasoning from Dense References](https://mlsanity.com/paper/2610.10893/) (2026)

## Abstract (arXiv)

Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .