mlsanityData to 7 Oct 2026

CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs

arXiv 2408.10433 ↗First posted 19 Aug 2024cs.CVcited by 49

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 86 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

64 of 86 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  2. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  4. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  5. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  6. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023Mistral 7Bcited by 3,065
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  13. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  14. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  15. 2019Towards VQA Models That Can Readcited by 1,696
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  17. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  18. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  19. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  20. 2023LIMA: Less Is More for Alignmentcited by 1,080
  21. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  24. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  25. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  26. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  27. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  28. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  29. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  30. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  31. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  32. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  33. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  34. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  35. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  36. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  37. 2023Data Filtering Networkscited by 247
  38. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 29; 2026: 14.

49 papers.

Sort:

By area:

  1. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  4. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  5. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  6. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  7. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  8. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  9. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  10. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  11. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  12. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decodingcited by 12
37 more
  1. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  2. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  3. 2024Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approachcited by 9
  4. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  5. 2025AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animationcited by 7
  6. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  7. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  8. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 7
  9. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignmentcited by 6
  10. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 6
  11. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  12. 2025Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinationscited by 5
  13. 2025NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimationcited by 5
  14. 2025Spatial Preference Rewarding for MLLMs Spatial Understandingcited by 3
  15. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinationscited by 3
  16. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 3
  17. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 3
  18. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  19. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  20. 2025Energy-Guided Decoding for Object Hallucination Mitigationcited by 2
  21. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  22. 2026VISTA: Enhancing Visual Conditioning via Track-Following Preference Optimization in Vision-Language-Action Modelscited by 2
  23. 2026Mitigating Multimodal Hallucination via Phase-wise Self-rewardcited by 2
  24. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  25. 2024EVLM: Self-Reflective Multimodal Reasoning for Cross-Dimensional Visual Editingcited by 1
  26. 2025Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMscited by 1
  27. 2025Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generationcited by 1
  28. 2025Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimizationcited by 1
  29. 2026Benchmarking Direct Preference Optimization for Medical Large Vision-Language Modelscited by 1
  30. 2026Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMscited by 1
  31. 2026Delineating Knowledge Boundaries for Honest Large Vision-Language Modelscited by 1
  32. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impactcited by 0
  33. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  34. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  35. 2026Learning to Rank Caption Chains for Video-Text Alignmentcited by 0
  36. 2026Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reductioncited by 0
  37. 2026Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization
  2. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key
  3. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models
  4. 2025Mitigating Image Captioning Hallucinations in Vision-Language Models
  5. 2025Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs
  6. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
  7. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing
  8. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence