mlsanityData to 7 Oct 2026

DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO

arXiv 2506.07464 ↗First posted 9 Jun 2025cs.CV cs.AIcited by 57

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent works have demonstrated the effectiveness of reinforcement learning (RL)-based post-training for enhancing the reasoning capabilities of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) has shown impressive success using a PPO-style reinforcement learning algorithm with group-normalized rewards. However, the effectiveness of GRPO in Video Large Language Models (VideoLLMs) remains underexplored. In this paper, we explore GRPO and identify two issues that hinder effective learning: (1) reliance on safeguards, and (2) vanishing advantage. To mitigate these challenges, we propose DeepVideo-R1, a video large language model trained with Reg-GRPO (Regressive GRPO) and difficulty-aware data augmentation. Reg-GRPO reformulates the GRPO loss function as a regression task that directly predicts the advantage in GRPO, eliminating the need for safeguards such as clipping and min operations. This directly aligns the model with the advantages, providing guidance to prefer better outputs. The difficulty-aware data augmentation strategy augments input prompts/videos to target solvable difficulty levels, enabling diverse reward signals. Our experimental results show that our approach significantly improves video reasoning performance across multiple benchmarks.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 81 of its 99 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

81 of 99 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Boosting MLLM Reasoning with Text-Debiased Hint-GRPO31 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
38 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2017Dense-Captioning Events in Videoscited by 789
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  12. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  13. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  14. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  15. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  16. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  17. 2020Rescaling Egocentric Visioncited by 529
  18. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  19. 2024Long Context Transfer from Language to Visioncited by 502
  20. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  21. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  22. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  23. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  24. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  25. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  26. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  27. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  28. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  29. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  30. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  31. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  32. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 225
  33. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  34. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  35. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  36. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  37. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  38. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 34.

57 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  3. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  4. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  5. 2025Nudging the Boundaries of LLM Reasoningcited by 25
  6. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
  7. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesiscited by 22
  8. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 20
  9. 2026VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twicecited by 19
  10. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  11. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  12. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
38 more
  1. 2025VIDEOP2R: Video Understanding from Perception to Reasoningcited by 8
  2. 2025VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotationscited by 7
  3. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  4. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 7
  5. 2026VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videoscited by 6
  6. 2025Representation Shift: Unifying Token Compression with FlashAttentioncited by 5
  7. 2025TimeScope: Towards Task-Oriented Temporal Grounding In Long Videoscited by 5
  8. 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learningcited by 5
  9. 2026TeleBoost: A Systematic Alignment Framework for High-Fidelity, Controllable, and Robust Video Generationcited by 5
  10. 2026SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agentscited by 5
  11. 2025TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusingcited by 3
  12. 2025VideoCuRL: Video Curriculum Reinforcement Learning with Orthogonal Difficulty Decompositioncited by 3
  13. 2026DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantagecited by 3
  14. 2026Motion-o: Trajectory-Grounded Video Reasoningcited by 3
  15. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  16. 2026Temporal-Aware Reasoning Optimization for Video Temporal Groundingcited by 3
  17. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  18. 2025Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understandingcited by 2
  19. 2026APPO: Attention-guided Perception Policy Optimization for Video Reasoningcited by 2
  20. 2026STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answeringcited by 2
  21. 2026EvoVid: Temporal-Centric Self-Evolution for Video Large Language Modelscited by 2
  22. 2025Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimizationcited by 1
  23. 2025Anatomy-R1: Enhancing Anatomy Reasoning in Multimodal Large Language Models via Anatomical Similarity Curriculum and Group Diversity Augmentationcited by 1
  24. 2026MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Predictioncited by 1
  25. 2026Weaver: End-to-End Agentic System Training for Video Interleaved Reasoningcited by 1
  26. 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinkingcited by 1
  27. 2026HORNet: Task-Guided Frame Selection for Video Question Answering with Vision-Language Modelscited by 1
  28. 2026Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learningcited by 1
  29. 2026Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMscited by 1
  30. 2026Confidence-Aware Tool Orchestration for Robust Video Understandingcited by 1
  31. 2026VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinementcited by 1
  32. 2026I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoningcited by 1
  33. 2026Post-Training VLMs for Video Mistake Detectioncited by 1
  34. 2025Decoupling Perception from Reasoning for Hallucination-Resistant Video Understandingcited by 0
  35. 2026Improving Large Molecular Language Model via Relation-aware Multimodal Collaborationcited by 0
  36. 2026Consensus Group Relative Policy Optimization for Text Generationcited by 0
  37. 2026From Imitation to Intuition: Intrinsic Reasoning for Open-Instance Video Classificationcited by 0
  38. 2026ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Rewardcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  2. 2026GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning
  3. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models
  4. 2025VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization
  5. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  6. 2026GRPO Training Dynamics for Small Language Models
  7. 2026GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings
  8. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping