mlsanityData to 7 Oct 2026

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

arXiv 2605.04641 ↗First posted 6 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 56 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

46 of 56 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation8 Nov
  2. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigation3 Jan
  3. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models22 Nov
  4. 2024CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs19 Nov
  5. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering21 Oct
  6. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality7 Oct
  7. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  8. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  9. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models30 Jun
  10. 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models16 Jun
  11. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  12. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization24 May
12 more
  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  2. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  9. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  10. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
10 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  2. 2018Object Hallucination in Image Captioningcited by 661
  3. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  4. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  5. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  6. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  7. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 34
  8. 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
  9. 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questionscited by 14
  10. 2025ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  2. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  3. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
  4. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
  5. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  6. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  7. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  8. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations