mlsanityData to 14 Oct 2026

Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration

arXiv 2505.21472 ↗First posted 27 May 2025cs.CV cs.CL cs.MMcited by 11

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVLMs) achieve impressive performance on multimodal tasks but often suffer from hallucination, and confidently describe objects or attributes not present in the image. Current training-free interventions struggle to maintain accuracy in open-ended and long-form generation scenarios. We introduce the Confidence-Aware Attention Calibration (CAAC) framework to address this challenge by targeting two key biases: spatial perception bias, which distributes attention disproportionately across image tokens, and modality bias, which shifts focus from visual to textual inputs over time. CAAC employs a two-step approach: Visual-Token Calibration (VTC) to balance attention across visual tokens, and Adaptive Attention Re-Scaling (AAR) to reinforce visual grounding guided by the model's confidence. This confidence-driven adjustment ensures consistent visual alignment during generation. Experiments on CHAIR, AMBER, and POPE benchmarks demonstrate that CAAC outperforms baselines, particularly in long-form generations, effectively reducing hallucination.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 37 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

24 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding1 Mar
  2. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs15 Nov
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  4. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  5. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  9. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  10. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
3 more
  1. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  2. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  3. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoningcited by 8
  2. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  3. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 4
  4. 2025Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddingscited by 3
  5. 2026Context-Aware Decoding for Faithful Vision-Language Generationcited by 1
  6. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  7. 2026Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrievalcited by 0
  8. 2026Attending to Multimodal Generation One Token at a Timecited by 0
  9. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
  10. 2026Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agentscited by 0
  11. 2026HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decidecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration
  2. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
  3. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  4. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  5. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
  6. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
  7. 2026HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models
  8. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models