mlsanityData to 14 Oct 2026

DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models

arXiv 2411.18659 ↗First posted 27 Nov 2024cs.CV cs.AIcited by 11

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVLMs) have demonstrated exceptional performance on complex multimodal tasks. However, they continue to suffer from significant hallucination issues, including object, attribute, and relational hallucinations. To accurately detect these hallucinations, we investigated the variations in cross-modal attention patterns between hallucination and non-hallucination states. Leveraging these distinctions, we developed a lightweight detector capable of identifying hallucinations. Our proposed method, Detecting Hallucinations by Cross-modal Attention Patterns (DHCP), is straightforward and does not require additional LVLM training or extra LVLM inference steps. Experimental results show that DHCP achieves remarkable performance in hallucination detection. By offering novel insights into the identification and analysis of hallucinations in LVLMs, DHCP contributes to advancing the reliability and trustworthiness of these models. The code is available at https://github.com/btzyd/DHCP.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 45 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

25 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  4. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  5. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
8 more
  1. 2018Object Hallucination in Image Captioningcited by 669
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  3. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  4. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  5. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  6. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  7. 2024Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned Modelscited by 15
  8. 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questionscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  2. 2025Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinkingcited by 6
  3. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
  4. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  5. 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Modelscited by 4
  6. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Headscited by 4
  7. 2025When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQAcited by 3
  8. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinationscited by 3
  9. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  10. 2026Metaphor Is Not All Attention Needscited by 0
  11. 2026A Benchmark for Hallucination Detection in VLMs for Gastrointestinal Endoscopycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
  2. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context
  3. 2025PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models
  4. 2026UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space
  5. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
  6. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation
  7. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  8. 2024Reference-free Hallucination Detection for Large Vision-Language Models