# CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs

arXiv: https://arxiv.org/abs/2408.10433 · HTML: https://mlsanity.com/paper/2408.10433/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 64 of its 86 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 64)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)

## Built on by (most cited first, 30 of 49)

- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model](https://mlsanity.com/paper/2501.12368/) (2025)
- [MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models](https://mlsanity.com/paper/2410.17637/) (2024)
- [Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability](https://mlsanity.com/paper/2501.01346/) (2025)
- [Mitigating Object Hallucinations via Sentence-Level Early Intervention](https://mlsanity.com/paper/2507.12455/) (2025)
- [Identifying and Mitigating Position Bias of Multi-image Vision-Language Models](https://mlsanity.com/paper/2503.13792/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization](https://mlsanity.com/paper/2411.10436/) (2024)
- [PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning](https://mlsanity.com/paper/2504.05810/) (2025)
- [Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization](https://mlsanity.com/paper/2506.11712/) (2025)
- [Aligning Multimodal LLM with Human Preference: A Survey](https://mlsanity.com/paper/2503.14504/) (2025)
- [Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding](https://mlsanity.com/paper/2505.17529/) (2025)
- [TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention](https://mlsanity.com/paper/2503.10602/) (2025)
- [Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding](https://mlsanity.com/paper/2503.10183/) (2025)
- [Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach](https://mlsanity.com/paper/2411.17760/) (2024)
- [More Images, More Problems? A Controlled Analysis of VLM Failure Modes](https://mlsanity.com/paper/2601.07812/) (2026)
- [AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization](https://mlsanity.com/paper/2504.01735/) (2025)
- [SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards](https://mlsanity.com/paper/2506.07963/) (2025)
- [AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation](https://mlsanity.com/paper/2506.11144/) (2025)
- [ChartCap: Mitigating Hallucination of Dense Chart Captioning](https://mlsanity.com/paper/2508.03164/) (2025)
- [SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense](https://mlsanity.com/paper/2510.16596/) (2025)
- [Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment](https://mlsanity.com/paper/2412.17417/) (2024)
- [Self-Improvement in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2510.02665/) (2025)
- [MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization](https://mlsanity.com/paper/2603.03192/) (2026)
- [Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations](https://mlsanity.com/paper/2509.11287/) (2025)
- [NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation](https://mlsanity.com/paper/2510.21122/) (2025)
- [Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs](https://mlsanity.com/paper/2602.15556/) (2026)
- [Spatial Preference Rewarding for MLLMs Spatial Understanding](https://mlsanity.com/paper/2510.14374/) (2025)
- [DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations](https://mlsanity.com/paper/2601.00623/) (2026)
- [VISTA: Enhancing Visual Conditioning via Track-Following Preference Optimization in Vision-Language-Action Models](https://mlsanity.com/paper/2602.05049/) (2026)

## Abstract (arXiv)

Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved.