mlsanityData to 7 Oct 2026

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

arXiv 2501.09695 ↗First posted 16 Jan 2025cs.CVcited by 64

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Hallucination remains a major challenge for Large Vision-Language Models (LVLMs). Direct Preference Optimization (DPO) has gained increasing attention as a simple solution to hallucination issues. It directly learns from constructed preference pairs that reflect the severity of hallucinations in responses to the same prompt and image. Nonetheless, different data construction methods in existing works bring notable performance variations. We identify a crucial factor here: outcomes are largely contingent on whether the constructed data aligns on-policy w.r.t the initial (reference) policy of DPO. Theoretical analysis suggests that learning from off-policy data is impeded by the presence of KL-divergence between the updated policy and the reference policy. From the perspective of dataset distribution, we systematically summarize the inherent flaws in existing algorithms that employ DPO to address hallucination issues. To alleviate the problems, we propose On-Policy Alignment (OPA)-DPO framework, which uniquely leverages expert feedback to correct hallucinated responses and aligns both the original and expert-revised responses in an on-policy manner. Notably, with only 4.8k data, OPA-DPO achieves an additional reduction in the hallucination rate of LLaVA-1.5-7B: 13.26% on the AMBER benchmark and 5.39% on the Object-Hal benchmark, compared to the previous SOTA algorithm trained with 16k samples. Our implementation is available at https://github.com/zhyang2226/OPA-DPO.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 64 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

48 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  2. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding30 Sep
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2024Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models2 Feb
  6. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  11. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
24 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  13. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  14. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  15. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  16. 2024Iterative Reasoning Preference Optimizationcited by 244
  17. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  18. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  19. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  20. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  21. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 72
  22. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  23. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediationscited by 11
  24. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Groundingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 43.

64 papers.

Sort:

By area:

  1. 2025HunyuanVideo 1.5 Technical Reportcited by 148
  2. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  3. 2025Learning Dynamics of VLM Finetuningcited by 17
  4. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  5. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  6. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 7
  7. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigationcited by 7
  8. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  9. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  10. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
  11. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
  12. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
38 more
  1. 2026Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimizationcited by 5
  2. 2026Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppressioncited by 5
  3. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  4. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  5. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  6. 2025RL makes MLLMs see better than SFTcited by 3
  7. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3
  8. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 3
  9. 2026Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Modelscited by 3
  10. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 3
  11. 2026Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Modelscited by 3
  12. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 3
  13. 2025SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winnercited by 2
  14. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  15. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  16. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflectioncited by 2
  17. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
  18. 2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
  19. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  20. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  21. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  22. 2025Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMscited by 1
  23. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendationcited by 1
  24. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  25. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  26. 2026GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinationscited by 1
  27. 2026EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectorscited by 1
  28. 2026Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMscited by 1
  29. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimizationcited by 1
  30. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  31. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  32. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  33. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  34. 2026Learning to Rank Caption Chains for Video-Text Alignmentcited by 0
  35. 2026HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Modelscited by 0
  36. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  37. 2026Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimizationcited by 0
  38. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibrationcited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization
  2. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  3. 2025Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations
  4. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
  5. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization
  6. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  7. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  8. 2025Stop learning it all to mitigate visual hallucination, Focus on the hallucination target