mlsanityData to 14 Oct 2026

ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models

arXiv 2411.15268 ↗First posted 22 Nov 2024cs.CV cs.CLcited by 30

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite the recent breakthroughs achieved by Large Vision Language Models (LVLMs) in understanding and responding to complex visual-textual contexts, their inherent hallucination tendencies limit their practical application in real-world scenarios that demand high levels of precision. Existing methods typically either fine-tune the LVLMs using additional data, which incurs extra costs in manual annotation and computational resources or perform comparisons at the decoding stage, which may eliminate useful language priors for reasoning while introducing inference time overhead. Therefore, we propose ICT, a lightweight, training-free method that calculates an intervention direction to shift the model's focus towards different levels of visual information, enhancing its attention to high-level and fine-grained visual details. During the forward pass stage, the intervention is applied to the attention heads that encode the overall image information and the fine-grained object details, effectively mitigating the phenomenon of overly language priors, and thereby alleviating hallucinations. Extensive experiments demonstrate that ICT achieves strong performance with a small amount of data and generalizes well across different datasets and models. Our code will be public.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 86 of its 98 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

86 of 98 references.

Closest ancestors · 26

Same direction, newest first

  1. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning30 Aug
  2. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  3. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation1 Aug
  4. 2024VACoDe: Visual Augmented Contrastive Decoding26 Jul
  5. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models30 Jun
  6. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  7. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models4 Jun
  8. 2024NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models30 May
  9. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization24 May
  10. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  11. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  12. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset17 Mar
14 more
  1. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  2. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  3. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  9. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  10. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  11. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  12. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  13. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  14. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  13. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  14. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  15. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  16. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 260
  17. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  18. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  19. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  20. 2022Large Language Models with Controllable Working Memorycited by 169
  21. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  22. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  23. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  24. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  25. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  26. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  27. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
  28. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  29. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  30. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  31. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 67
  32. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
  33. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  34. 2023Nationality Bias in Text Generationcited by 50
  35. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  36. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 45
  37. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 44
  38. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgerycited by 39

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 17.

30 papers.

Sort:

By area:

  1. 2025Steer LLM Latents for Hallucination Detectioncited by 50
  2. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  3. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 23
  4. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  5. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 10
  6. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  7. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  8. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  9. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  10. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  11. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 5
  12. 2026Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppressioncited by 5
18 more
  1. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 4
  2. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  3. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  4. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  5. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  6. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gatingcited by 2
  7. 2025Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Modelscited by 1
  8. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attributioncited by 1
  9. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMscited by 1
  10. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
  11. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  12. 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Modelscited by 0
  13. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steeringcited by 0
  14. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steeringcited by 0
  15. 2026Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verificationcited by 0
  16. 2026Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methodscited by 0
  17. 2026The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMscited by 0
  18. 2026Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  2. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  3. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  4. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
  5. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  6. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  7. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
  8. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective