mlsanityData to 14 Oct 2026

Reducing Hallucinations in Vision-Language Models via Latent Space Steering

arXiv 2410.15778 ↗First posted 21 Oct 2024cs.CV cs.AI cs.LGcited by 59

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucination poses a challenge to the deployment of large vision-language models (LVLMs) in applications. Unlike in large language models (LLMs), hallucination in LVLMs often arises from misalignments between visual inputs and textual outputs. This paper investigates the underlying mechanisms of hallucination, focusing on the unique structure of LVLMs that distinguishes them from large language models (LLMs). We identify that hallucinations often arise from the sensitivity of text decoders to vision inputs, a natural phenomenon when image encoders and text decoders are pre-trained separately. Inspired by this, we introduce Visual and Textual Intervention (VTI), a novel technique designed to reduce hallucinations by steering latent space representations during inference to enhance the stability of vision features. As a task-agnostic test-time intervention, VTI can be easily applied to any problem without additional cost. Extensive experiments demonstrate that it can effectively reduce hallucinations and outperform baseline methods across multiple metrics, highlighting the critical role of vision feature stability in LVLMs.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 65 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

58 of 65 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  2. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination21 Mar
  3. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  9. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  11. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
4 more
  1. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
30 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  7. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  8. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  12. 2018Object Hallucination in Image Captioningcited by 669
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  14. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  15. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  16. 2022UL2: Unifying Language Learning Paradigmscited by 295
  17. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  18. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  19. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  20. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  21. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  22. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 64
  23. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steeringcited by 50
  24. 2022Avoiding spurious correlations via logit correctioncited by 32
  25. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitaskscited by 20
  26. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  27. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  28. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16
  29. 2024Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answeringcited by 14
  30. 2024Automated radiotherapy treatment planning guided by GPT-4Visioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 31; 2026: 26.

59 papers.

Sort:

By area:

  1. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  2. 2025Steer LLM Latents for Hallucination Detectioncited by 50
  3. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
  4. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucinationcited by 35
  5. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 32
  6. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  7. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  8. 2024Large Multi-modal Models Can Interpret Features in Large Multi-modal Modelscited by 23
  9. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 23
  10. 2025Auto-Slides: An Interactive Multi-Agent System for Creating and Customizing Research Presentationscited by 19
  11. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 16
  12. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
38 more
  1. 2025Learning to Steer: Input-dependent Steering for Multimodal LLMscited by 15
  2. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspectivecited by 12
  3. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  4. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  5. 2026Hallucination Begins Where Saliency Dropscited by 9
  6. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lenscited by 8
  7. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
  8. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigationcited by 7
  9. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  10. 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
  11. 2026Pixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervisioncited by 5
  12. 2025Concept Lancet: Image Editing with Compositional Representation Transplantcited by 4
  13. 2025DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Modelscited by 4
  14. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  15. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  16. 2025Towards Reasoning-Preserving Unlearning in Multimodal Large Language Modelscited by 4
  17. 2026Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Modelscited by 4
  18. 2026Efficient Reasoning with Balanced Thinkingcited by 4
  19. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  20. 2024Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?cited by 3
  21. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 3
  22. 2025Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysiscited by 3
  23. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 3
  24. 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
  25. 2026Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Modelscited by 3
  26. 2025ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Modelscited by 2
  27. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  28. 2025Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Modelscited by 2
  29. 2026TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMscited by 2
  30. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  31. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Frameworkcited by 2
  32. 2025GUIDE: Towards Scalable Advising for Research Ideascited by 1
  33. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  34. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  35. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editingcited by 1
  36. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steeringcited by 0
  37. 2026Causal Probing for Internal Visual Representations in Multimodal Large Language Modelscited by 0
  38. 2026Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flowcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  2. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
  3. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
  4. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  5. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  6. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  7. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  8. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models