mlsanityData to 7 Oct 2026

Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention

arXiv 2406.12718 ↗First posted 18 Jun 2024cs.CV cs.AI cs.CLcited by 74

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite great success across various multimodal tasks, Large Vision-Language Models (LVLMs) often encounter object hallucinations with generated textual responses being inconsistent with the actual objects in images. We examine different LVLMs and pinpoint that one root cause of object hallucinations lies with deficient attention on discriminative image features. Specifically, LVLMs often predominantly attend to prompt-irrelevant global features instead of prompt-relevant local features, undermining their visual grounding capacity and leading to object hallucinations. We propose Assembly of Global and Local Attention (AGLA), a training-free and plug-and-play approach that mitigates hallucinations by assembling global features for response generation and local features for visual discrimination simultaneously. Specifically, we introduce an image-prompt matching scheme that captures prompt-relevant local features from images, leading to an augmented view of the input image where prompt-relevant content is highlighted while irrelevant distractions are suppressed. Hallucinations can thus be mitigated with a calibrated logit distribution that is from generative global features of the original image and discriminative local features of the augmented image. Extensive experiments show the superiority of AGLA in LVLM hallucination mitigation, demonstrating its wide applicability across both discriminative and generative tasks. Our code is available at https://github.com/Lackel/AGLA.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 76 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

59 of 76 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  2. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  3. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  7. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  8. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  9. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  10. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  11. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
34 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  15. 2018Object Hallucination in Image Captioningcited by 661
  16. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  20. 2022UL2: Unifying Language Learning Paradigmscited by 292
  21. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  22. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  23. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  24. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  25. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  26. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  27. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  28. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  29. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  30. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitaskscited by 19
  31. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  32. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16
  33. 2024Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answeringcited by 14
  34. 2024DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 25; 2026: 42.

74 papers.

Sort:

By area:

  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  2. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 49
  3. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  4. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  5. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  6. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  7. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decodingcited by 14
  8. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  9. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
  10. 2025Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoningcited by 8
  11. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 8
  12. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
38 more
  1. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventionscited by 6
  2. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  3. 2026DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Modelscited by 6
  4. 2024DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmarkcited by 5
  5. 2025FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoningcited by 5
  6. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  7. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluationcited by 5
  8. 2026Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppressioncited by 5
  9. 2024VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Modelscited by 4
  10. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Modelscited by 4
  11. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  12. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Modelscited by 4
  13. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  14. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  15. 2025Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddingscited by 3
  16. 2026Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decodingcited by 3
  17. 2026Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Modelscited by 3
  18. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 3
  19. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  20. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 3
  21. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 3
  22. 2025SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revisioncited by 2
  23. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinementcited by 2
  24. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentationcited by 2
  25. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  26. 2025CLASH: A Benchmark for Cross-Modal Contradiction Detectioncited by 2
  27. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 2
  28. 2026EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Modelscited by 2
  29. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMcited by 2
  30. 2026Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigationcited by 2
  31. 20263D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decodingcited by 2
  32. 2026Mitigating Multimodal Hallucination via Phase-wise Self-rewardcited by 2
  33. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Modelscited by 2
  34. 2026SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineeringcited by 2
  35. 2025Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Modelscited by 1
  36. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  37. 2025Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Modelscited by 1
  38. 2025PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Modelscited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  2. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  3. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration
  4. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  5. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  6. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models
  7. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
  8. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities