mlsanityData to 7 Oct 2026

Aligning Modalities in Vision Large Language Models via Preference Fine-tuning

arXiv 2402.11411 ↗First posted 18 Feb 2024cs.LG cs.CL cs.CVcited by 197

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Instruction-following Vision Large Language Models (VLLMs) have achieved significant progress recently on a variety of tasks. These approaches merge strong pre-trained vision models and large language models (LLMs). Since these components are trained separately, the learned representations need to be aligned with joint training on additional image-language pairs. This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations. In this work, we frame the hallucination problem as an alignment issue, tackle it with preference tuning. Specifically, we propose POVID to generate feedback data with AI models. We use ground-truth instructions as the preferred response and a two-stage approach to generate dispreferred data. First, we prompt GPT-4V to inject plausible hallucinations into the correct answer. Second, we distort the image to trigger the inherent hallucination behavior of the VLLM. This is an automated approach, which does not rely on human data generation or require a perfect expert, which makes it easily scalable. Finally, both of these generation strategies are integrated into an RLHF pipeline via Direct Preference Optimization. In experiments across broad benchmarks, we show that we can not only reduce hallucinations, but improve model performance across standard benchmarks, outperforming prior approaches. Our data and code are available at https://github.com/YiyangZhou/POVID.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 45 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

38 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  3. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2015VQA: Visual Question Answeringcited by 2,489
22 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  13. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  14. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  15. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  16. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  17. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  18. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  19. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  20. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  21. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  22. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 62; 2025: 95; 2026: 40.

197 papers.

Sort:

By area:

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  3. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  8. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  9. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  10. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  11. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  12. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
38 more
  1. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  2. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  3. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  4. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  5. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  6. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  7. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  8. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  9. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  10. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  11. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  12. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  13. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  14. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  15. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  16. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 62
  17. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  18. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  19. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  20. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  21. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  22. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  23. 2024Probing Visual Language Priors in VLMscited by 44
  24. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  25. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 39
  26. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  27. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  28. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  29. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  30. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  31. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  32. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 31
  33. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  34. 2024Automated Multi-level Preference for MLLMscited by 29
  35. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  36. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  37. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  38. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28

50 most cited of 197 All 197 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Token-Level Inference-Time Alignment for Vision-Language Models
  2. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  3. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models
  4. 2026Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization
  5. 2025Attention Guided Alignment in Efficient Vision-Language Models
  6. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
  8. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models