mlsanityData to 14 Oct 2026

Automated Multi-level Preference for MLLMs

arXiv 2405.11165 ↗First posted 18 May 2024cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Current multimodal Large Language Models (MLLMs) suffer from ``hallucination'', occasionally generating responses that are not grounded in the input images. To tackle this challenge, one promising path is to utilize reinforcement learning from human feedback (RLHF), which steers MLLMs towards learning superior responses while avoiding inferior ones. We rethink the common practice of using binary preferences (i.e., superior, inferior), and find that adopting multi-level preferences (e.g., superior, medium, inferior) is better for two benefits: 1) It narrows the gap between adjacent levels, thereby encouraging MLLMs to discern subtle differences. 2) It further integrates cross-level comparisons (beyond adjacent-level comparisons), thus providing a broader range of comparisons with hallucination examples. To verify our viewpoint, we present the Automated Multi-level Preference (AMP) framework for MLLMs. To facilitate this framework, we first develop an automated dataset generation pipeline that provides high-quality multi-level preference datasets without any human annotators. Furthermore, we design the Multi-level Direct Preference Optimization (MDPO) algorithm to robustly conduct complex multi-level preference learning. Additionally, we propose a new hallucination benchmark, MRHal-Bench. Extensive experiments across public hallucination and general benchmarks, as well as our MRHal-Bench, demonstrate the effectiveness of our proposed method. Code is available at https://github.com/takomc/amp.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 46 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

41 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
24 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  13. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  14. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  15. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  16. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  17. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  18. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  19. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  20. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  21. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  22. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  23. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  24. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 7.

29 papers.

Sort:

By area:

  1. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  2. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  3. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  5. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  7. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  8. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  9. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  10. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
  11. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
  12. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimizationcited by 6
17 more
  1. 2025Can Large Models Fool the Eye? A New Turing Test for Biological Animationcited by 6
  2. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  3. 2025Factorized Learning for Temporally Grounded Video-Language Modelscited by 4
  4. 2025EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgerycited by 3
  5. 2025Spatial Preference Rewarding for MLLMs Spatial Understandingcited by 3
  6. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  7. 2025Stop learning it all to mitigate visual hallucination, Focus on the hallucination targetcited by 2
  8. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Modelscited by 1
  9. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
  10. 2026Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understandingcited by 1
  11. 2025Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflectioncited by 0
  12. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  13. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  14. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigationcited by 0
  15. 2026Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMscited by 0
  16. 2026Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generationcited by 0
  17. 2026Fine-Grained Multi Image Object Hallucination Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
  2. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  3. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
  4. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  5. 2026DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations
  6. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Study
  7. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  8. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models