mlsanityData to 14 Oct 2026

Mitigating Multilingual Hallucination in Large Vision-Language Models

arXiv 2408.00550 ↗First posted 1 Aug 2024cs.CV cs.AI cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

While Large Vision-Language Models (LVLMs) have exhibited remarkable capabilities across a wide range of tasks, they suffer from hallucination problems, where models generate plausible yet incorrect answers given the input image-query pair. This hallucination phenomenon is even more severe when querying the image in non-English languages, while existing methods for mitigating hallucinations in LVLMs only consider the English scenarios. In this paper, we make the first attempt to mitigate this important multilingual hallucination in LVLMs. With thorough experiment analysis, we found that multilingual hallucination in LVLMs is a systemic problem that could arise from deficiencies in multilingual capabilities or inadequate multimodal abilities. To this end, we propose a two-stage Multilingual Hallucination Removal (MHR) framework for LVLMs, aiming to improve resistance to hallucination for both high-resource and low-resource languages. Instead of relying on the intricate manual annotations of multilingual resources, we fully leverage the inherent capabilities of the LVLM and propose a novel cross-lingual alignment method, which generates multiple responses for each image-query input and then identifies the hallucination-aware pairs for each language. These data pairs are finally used for direct preference optimization to prompt the LVLMs to favor non-hallucinating responses. Experimental results show that our MHR achieves a substantial reduction in hallucination generation for LVLMs. Notably, on our extended multilingual POPE benchmark, our framework delivers an average increase of 19.0% in accuracy across 13 different languages. Our code and model weights are available at https://github.com/ssmisya/MHR

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 64 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

44 of 64 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  6. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  7. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  9. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 35

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
23 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  2. 2018Object Hallucination in Image Captioningcited by 669
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  6. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  8. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  9. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  10. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  11. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  12. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  13. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  14. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcarecited by 62
  15. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  16. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  17. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  18. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  19. 2022Prompt-based Learning for Unpaired Image Captioningcited by 23
  20. 2024Timo: Towards Better Temporal Reasoning for Language Modelscited by 21
  21. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Frameworkcited by 20
  22. 2024Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Modelscited by 9
  23. 2023UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Promptscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  2. 2025From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibrationcited by 8
  3. 2026VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention
  2. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  3. 2024Hallucination of Multimodal Large Language Models: A Survey
  4. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  5. 2024A Survey on Hallucination in Large Vision-Language Models
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  7. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  8. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)