mlsanityData to 7 Oct 2026

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

arXiv 2402.03746 ↗First posted 6 Feb 2024cs.CVcited by 36

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs). The previous approaches for VLMMs involved Supervised Fine-Tuning (SFT) with instruction-tuned datasets, integrating LLM with visual encoders, and adding additional learnable modules. Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data compared to text-only data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF), providing self-preference feedback to refine itself and facilitating the alignment of video and text modalities. In specific, we propose context-aware reward modeling by providing detailed video descriptions as context during the generation of preference feedback in order to enrich the understanding of video content. Demonstrating enhanced performance across diverse video benchmarks, our multimodal RLAIF approach, VLM-RLAIF, outperforms existing approaches, including the SFT model. We commit to open-sourcing our code, models, and datasets to foster further research in this area.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 47 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

23 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
10 more
  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  2. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  3. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  4. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  5. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  6. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  7. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  8. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  9. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 23
  10. 2023VLM-Eval: A General Evaluation on Video Large Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 23; 2026: 7.

36 papers.

Sort:

By area:

  1. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  2. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  3. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  4. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  5. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  6. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  7. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 18
  8. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  9. 2025ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understandingcited by 15
  10. 2025Is Your Video Language Model a Reliable Judge?cited by 14
  11. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  12. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
24 more
  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  2. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  3. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  4. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  5. 2025SafeVid: Toward Safety Aligned Video Large Multimodal Modelscited by 9
  6. 2025Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flowcited by 9
  7. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  8. 2025Reflection-Based Task Adaptation for Self-Improving VLAcited by 8
  9. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  10. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  11. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 5
  12. 2026CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instructioncited by 5
  13. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  14. 2026VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generationcited by 3
  15. 2025What Happens When: Learning Temporal Orders of Events in Videoscited by 2
  16. 2025Stealing Creator's Workflow: A Creator-Inspired Agentic Framework with Iterative Feedback Loop for Improved Scientific Short-form Generationcited by 1
  17. 2025LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMscited by 1
  18. 2026Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluationcited by 1
  19. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 1
  20. 2024Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Modelscited by 0
  21. 2025Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedbackcited by 0
  22. 2025COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stabilitycited by 0
  23. 2026See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designerscited by 0
  24. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Multimodal Knowledge Alignment with Reinforcement Learning
  2. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency
  3. 2025Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
  4. 2025InstructEngine: Instruction-driven Text-to-Image Alignment
  5. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
  6. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1
  7. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs
  8. 2025Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning