mlsanityData to 7 Oct 2026

Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward

arXiv 2404.01258 ↗First posted 1 Apr 2024cs.CV cs.AIcited by 145

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has not been conclusively established. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying this tailored reward through DPO significantly improves the performance of video LMMs on video QA tasks.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 63 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

37 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback6 Feb
  2. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
20 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  3. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  7. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  8. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  9. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  10. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  11. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  12. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  13. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  14. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  15. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  16. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  17. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  18. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  19. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  20. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 74; 2026: 30.

145 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  3. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  4. 2024Long Context Transfer from Language to Visioncited by 502
  5. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  6. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  7. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  9. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  10. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  11. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  12. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
38 more
  1. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  2. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  3. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  4. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  5. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  6. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  7. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  8. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  9. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  10. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  11. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  12. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  13. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  14. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  15. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  16. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 48
  17. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  18. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  19. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  20. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  21. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  22. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  23. 2024Temporal Reasoning Transfer from Text to Videocited by 36
  24. 2025A Survey of Direct Preference Optimizationcited by 35
  25. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  26. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  27. 2024Self-Supervised Visual Preference Alignmentcited by 31
  28. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  29. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  30. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  31. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  32. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  33. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  34. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  35. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  36. 2024STIV: Scalable Text and Image Conditioned Video Generationcited by 24
  37. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  38. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Models
  2. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understanding
  3. 2025Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization
  4. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
  5. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
  6. 2025ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
  7. 2025M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following
  8. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models