mlsanityData to 14 Oct 2026

Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment

arXiv 2405.18654 ↗First posted 28 May 2024cs.CVcited by 42

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite their significant advancements, Multimodal Large Language Models (MLLMs) often generate factually inaccurate information, referred to as hallucination. In this work, we address object hallucinations in MLLMs, where information is generated about an object not present in the input image. We introduce Data-augmented Phrase-level Alignment (DPA), a novel loss which can be applied to instruction-tuned off-the-shelf MLLMs to mitigate hallucinations, while preserving their general vision-language capabilities. To fine-tune MLLMs with DPA, we first generate a set of `hallucinated' and `correct' response pairs through generative data augmentation by selectively altering the ground-truth information of the correct responses at a phrase level. The DPA loss is then used to train MLLMs to reduce the likelihood of hallucinated phrases compared to the correct ones. Our thorough evaluation on various benchmarks confirms the effectiveness of DPA in mitigating hallucination while retaining the out-of-the-box performance of the MLLMs on general tasks. For instance, MLLMs finetuned with DPA, which we refer to as Hallucination Attenuated Language and Vision Assistant (HALVA), improve F1 by up to 13.4% on hallucination visual question-answering and reduce the hallucination rate by up to 4.2% on image description tasks.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 28 May 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 23.

42 papers.

Sort:

By area:

  1. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  2. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  3. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  4. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  5. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  6. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  7. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  8. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  9. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  10. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  11. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  12. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 5
30 more
  1. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  2. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  3. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 4
  4. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflectioncited by 3
  5. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  6. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  7. 2025Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generationcited by 1
  8. 2025Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMscited by 1
  9. 2025VaPR -- Vision-language Preference alignment for Reasoningcited by 1
  10. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  11. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  12. 2026Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMscited by 1
  13. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1
  14. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  15. 2025CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Cloudscited by 0
  16. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  17. 2025Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decodingcited by 0
  18. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  19. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steeringcited by 0
  20. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  21. 2026Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimizationcited by 0
  22. 2026Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMscited by 0
  23. 2026Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilizationcited by 0
  24. 2026Fine-Grained Multi Image Object Hallucination Benchmarkcited by 0
  25. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  26. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Modelscited by 0
  27. 2026DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editingcited by 0
  28. 2026Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learningcited by 0
  29. 2026Improving OCR Faithfulness via Gated and Attenuated On-Policy Distillationcited by 0
  30. 2026MoLE: Mixture of Latent Experts for Complementary Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models
  2. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models
  4. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  5. 2024Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  7. 2026From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment
  8. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective