mlsanityData to 7 Oct 2026

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

arXiv 2406.11839 ↗First posted 17 Jun 2024cs.CV cs.AI cs.CLcited by 92

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. Through a comparative experiment, we identify the unconditional preference problem in multimodal preference optimization, where the model overlooks the image condition. To address this problem, we propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. Moreover, we introduce a reward anchor that forces the reward to be positive for chosen responses, thereby avoiding the decrease in their likelihood -- an intrinsic problem of relative preference optimization. Experiments on two multimodal LLMs of different sizes and three widely used benchmarks demonstrate that mDPO effectively addresses the unconditional preference problem in multimodal preference optimization and significantly improves model performance, particularly in reducing hallucination.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 51 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

31 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  2. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  3. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  4. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  6. 2018Object Hallucination in Image Captioningcited by 661
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  12. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
14 more
  1. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  2. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  3. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  4. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  5. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  6. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  7. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  8. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  9. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  10. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  11. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignmentcited by 42
  12. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  13. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
  14. 2024Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspectivecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 52; 2026: 32.

92 papers.

Sort:

By area:

  1. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  2. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  3. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  4. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  5. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  6. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  7. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  8. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  9. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27
  10. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  11. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  12. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
38 more
  1. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  2. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  3. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 18
  4. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  5. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  6. 2025ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understandingcited by 15
  7. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  8. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  9. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  10. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generationcited by 12
  11. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  12. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  13. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  14. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  15. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  16. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  17. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  18. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  19. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
  20. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  21. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  22. 2025Mol-LLM: Multimodal Generalist Molecular LLM with Improved Graph Utilizationcited by 6
  23. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimizationcited by 6
  24. 2026Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognitioncited by 6
  25. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  26. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Modelscited by 5
  27. 2025Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Modelscited by 5
  28. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  29. 2025Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaborationcited by 5
  30. 2026Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimizationcited by 5
  31. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  32. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  33. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  34. 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Modelscited by 3
  35. 2025Spatial Preference Rewarding for MLLMs Spatial Understandingcited by 3
  36. 2025RL makes MLLMs see better than SFTcited by 3
  37. 2025See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoningcited by 3
  38. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  2. 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
  3. 2024Multi-Reference Preference Optimization for Large Language Models
  4. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  5. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  6. 2024sDPO: Don't Use Your Data All at Once
  7. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
  8. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment