mlsanityData to 7 Oct 2026

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

arXiv 2609.00231 ↗First posted 31 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 33 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

21 of 33 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  2. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
  3. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering21 Oct
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  11. 2019Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Modelscited by 118
  12. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 37
1 more
  1. 2024PartImageNet++ Dataset: Scaling up Part-based Models for Robust Recognitioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
  2. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  3. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
  4. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  5. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediations
  6. 2025Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
  7. 2026ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation
  8. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations