mlsanityData to 14 Oct 2026

V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization

arXiv 2411.02712 ↗First posted 5 Nov 2024cs.CV cs.AIcited by 67

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Large vision-language models (LVLMs) suffer from hallucination, resulting in misalignment between the output textual response and the input visual content. Recent research indicates that the over-reliance on the Large Language Model (LLM) backbone, as one cause of the LVLM hallucination, inherently introduces bias from language priors, leading to insufficient context attention to the visual inputs. We tackle this issue of hallucination by mitigating such over-reliance through preference learning. We propose Vision-guided Direct Preference Optimization (V-DPO) to enhance visual context learning at training time. To interpret the effectiveness and generalizability of V-DPO on different types of training data, we construct a synthetic dataset containing both response- and image-contrast preference pairs, compared against existing human-annotated hallucination samples. Our approach achieves significant improvements compared with baseline methods across various hallucination benchmarks. Our analysis indicates that V-DPO excels in learning from image-contrast preference data, demonstrating its superior ability to elicit and understand nuances of visual context. Our code is publicly available at https://github.com/YuxiXie/V-DPO.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 39 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

29 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  2. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  5. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  6. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
10 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  2. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  3. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  4. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  5. 2023Stay on topic with Classifier-Free Guidancecited by 91
  6. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  7. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  8. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  9. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  10. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 34; 2026: 31.

67 papers.

Sort:

By area:

  1. 2024Probing Visual Language Priors in VLMscited by 45
  2. 2025A Survey of Direct Preference Optimizationcited by 36
  3. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  4. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  5. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  6. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  7. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  8. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 24
  9. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 17
  10. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 17
  11. 2025Learning Dynamics of VLM Finetuningcited by 17
  12. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 16
38 more
  1. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  2. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  3. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  4. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  5. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 10
  6. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  7. 2026RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoningcited by 8
  8. 2024Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigationcited by 7
  9. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimizationcited by 6
  10. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
  11. 2026Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Modelscited by 6
  12. 2025LPOI: Listwise Preference Optimization for Vision Language Modelscited by 5
  13. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  14. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  15. 2025VIBE: Can a VLM Read the Room?cited by 4
  16. 2025Multimedia Verification Through Multi-Agent Deep Research Multimodal Large Language Modelscited by 4
  17. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  18. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 4
  19. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  20. 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
  21. 2025RL makes MLLMs see better than SFTcited by 3
  22. 2025Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistencycited by 2
  23. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  24. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  25. 2025SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revisioncited by 2
  26. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  27. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  28. 2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
  29. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  30. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  31. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 2
  32. 2026Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillationcited by 2
  33. 2025LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMscited by 1
  34. 2026LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimizationcited by 1
  35. 2026ViewFusion: Structured Spatial Thinking Chains for Multi-View Reasoningcited by 1
  36. 2026Delineating Knowledge Boundaries for Honest Large Vision-Language Modelscited by 1
  37. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  38. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimizationcited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization
  2. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  3. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key
  4. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
  5. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  6. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  7. 2026Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization
  8. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence