mlsanityData to 7 Oct 2026

Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs

arXiv 2407.21771 ↗First posted 31 Jul 2024cs.CVcited by 174

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Existing Large Vision-Language Models (LVLMs) primarily align image features of vision encoder with Large Language Models (LLMs) to leverage their superior text generation capabilities. However, the scale disparity between vision encoder and language model may led to LLMs assuming a predominant role in multi-modal comprehension. This imbalance in LVLMs may result in the instances of hallucinatory. Concretely, LVLMs may generate consistent descriptions with or without visual input, indicating that certain outputs are influenced solely by context text. We refer to this phenomenon as "text inertia." To counteract this issue, we introduce a training-free algorithm to find an equilibrium point between image comprehension and language inference. Specifically, we adaptively involve adjusting and amplifying the attention weights assigned to image tokens, thereby granting greater prominence to visual elements. Meanwhile, we subtract the logits of multi-modal inputs from ones of pure text input, which can help LVLMs be not biased towards LLMs. By enhancing images tokens and reducing the stubborn output of LLM, we can let LVLM pay more attention to images, towards alleviating text inertia and reducing the hallucination in LVLMs. Our extensive experiments shows that this method substantially reduces the frequency of hallucinatory outputs in various LVLMs in terms of different metrics. Project page is available at https://lalbj.github.io/projects/PAI/.

Key facts. Cited by 174 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 43 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

41 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  4. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  6. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
21 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  4. 2023Vision Transformers Need Registerscited by 844
  5. 2023Steering Language Models With Activation Engineeringcited by 824
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2023Instruction Tuning with GPT-4cited by 676
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  13. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  14. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  15. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  16. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  17. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  18. 2023Stay on topic with Classifier-Free Guidancecited by 90
  19. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
  20. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  21. 2024SH2: Self-Highlighted Hesitation Helps You Decode More Truthfullycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 78; 2026: 83.

174 papers.

Sort:

By area:

  1. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  2. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  3. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  4. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  5. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  6. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  7. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  8. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  9. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  10. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  11. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  12. 2025MLLMs are Deeply Affected by Modality Biascited by 38
38 more
  1. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 37
  2. 2025ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLMcited by 36
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  4. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 27
  5. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  6. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  7. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  8. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  9. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
  10. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  11. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  12. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learningcited by 18
  13. 2024Have the VLMs Lost Confidence? A Study of Sycophancy in VLMscited by 15
  14. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  15. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMscited by 14
  16. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  17. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 13
  18. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  19. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 11
  20. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Interventioncited by 11
  21. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
  22. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  23. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 11
  24. 2025MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Modelscited by 10
  25. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  26. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 10
  27. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  28. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  29. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
  30. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Headscited by 9
  31. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 9
  32. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Modelscited by 8
  33. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  34. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  35. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  36. 2026VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneckcited by 8
  37. 2026Hallucination Begins Where Saliency Dropscited by 8
  38. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  2. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  3. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  4. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  5. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  6. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination
  7. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
  8. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance