mlsanityData to 7 Oct 2026

Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

arXiv 2405.17820 ↗First posted 28 May 2024cs.CV cs.AIcited by 46

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision Language Models (LVLMs) demonstrate strong capabilities in visual understanding and description, yet often suffer from hallucinations, attributing incorrect or misleading features to images. We observe that LVLMs disproportionately focus on a small subset of image tokens--termed blind tokens--which are typically irrelevant to the query (e.g., background or non-object regions). We hypothesize that such attention misalignment plays a key role in generating hallucinated responses. To mitigate this issue, we propose Attentional Vision Calibration (AvisC), a test-time approach that dynamically recalibrates the influence of blind tokens without modifying the underlying attention mechanism. AvisC first identifies blind tokens by analyzing layer-wise attention distributions over image tokens, then employs a contrastive decoding strategy to balance the influence of original and blind-token-biased logits. Experiments on standard benchmarks, including POPE, MME, and AMBER, demonstrate that AvisC effectively reduces hallucinations in LVLMs.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 50 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

42 of 50 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  2. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination21 Mar
  3. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  6. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  11. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  12. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
6 more
  1. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  2. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  6. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
12 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  9. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
  10. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  11. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Modelscited by 13
  12. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 24; 2026: 18.

46 papers.

Sort:

By area:

  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  2. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  3. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  4. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  5. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  6. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  7. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  8. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  9. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  10. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  11. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 18
  12. 2024Law of Vision Representation in MLLMscited by 16
34 more
  1. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMscited by 14
  2. 2024ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modelingcited by 9
  3. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  4. 2025SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decodingcited by 8
  5. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  6. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decodingcited by 6
  7. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  8. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  9. 2025ReCo: Reminder Composition Mitigates Hallucinations in Vision-Language Modelscited by 2
  10. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflectioncited by 2
  11. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentationcited by 2
  12. 2026EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Modelscited by 2
  13. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Modelscited by 2
  14. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  15. 2026SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineeringcited by 2
  16. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  17. 2025Concept Attractors in LLMs and their Applicationscited by 1
  18. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancementcited by 1
  19. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  20. 2026Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Frameworkcited by 1
  21. 2026Geometric Risk Control for Vision-Language Model OCRcited by 1
  22. 2026First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Modelscited by 1
  23. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attributioncited by 1
  24. 2025Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Modelscited by 0
  25. 2025CircuitProbe: Tracing Visual Temporal Evidence Flow in Video Language Modelscited by 0
  26. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Featurescited by 0
  27. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
  28. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  29. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0
  30. 2026Hallucination-aware intermediate representation edit in large vision-language modelscited by 0
  31. 2026Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidencecited by 0
  32. 2026Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignmentcited by 0
  33. 2026Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agentscited by 0
  34. 2026Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methodscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration
  2. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration
  3. 2025IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Models
  4. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  5. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  6. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model
  7. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  8. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction