mlsanityData to 14 Oct 2026

CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models

arXiv 2506.23590 ↗First posted 30 Jun 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Although Large Vision-Language Models (LVLMs) have demonstrated powerful capabilities in interpreting visual information, they frequently produce content that deviates from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly stronger when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-sensitive Attention Intervention (CAI), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern in response to caption queries to enhance LVLMs' visual perception capability. Extensive experimental results across four benchmarks covering both discriminative and generative tasks, demonstrate that CAI achieves state-of-the-art (SOTA) hallucination mitigating performance only with minimal additional inference cost.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 40 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

27 of 40 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigation3 Jan
  2. 2024CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs19 Nov
  3. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering21 Oct
  5. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality7 Oct
  6. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  8. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization24 May
  9. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  10. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  11. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  12. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
5 more
  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  4. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 10

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2018Object Hallucination in Image Captioningcited by 669
  7. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  10. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineeringcited by 8
  2. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Modelscited by 3
  3. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
  2. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models
  3. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  4. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  5. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  6. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
  7. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  8. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training