mlsanityData to 7 Oct 2026

Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models

arXiv 2603.16558 ↗First posted 17 Mar 2026cs.CV cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) achieve strong performance on many multimodal tasks, but object hallucinations severely undermine their reliability. Most existing studies focus on the text modality, attributing hallucinations to overly strong language priors and insufficient visual grounding. In contrast, we observe that abnormal attention patterns within the visual modality can also give rise to hallucinated objects. Building on this observation, we propose Segmentation-based Attention Entropy (SAE), which leverages semantic segmentation to quantify visual attention uncertainty in a semantic space. Based on SAE, we further design a reliability score for hallucination detection and an SAE-guided attention adjustment method that modifies visual attention at inference time to mitigate hallucinations. We evaluate our approach on public benchmarks and in real embodied multimodal scenarios with quadruped robots. Experiments show that SAE reduces hallucinations without additional training, improving LVLM reliability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 46 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

32 of 46 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow28 Feb
  2. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence18 Dec
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  4. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  5. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations3 Oct
  6. 2024Reference-free Hallucination Detection for Large Vision-Language Models11 Aug
  7. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)6 Aug
  8. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  9. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  10. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  11. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
3 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 17

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  10. 2018Object Hallucination in Image Captioningcited by 661
  11. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  12. 2017Beam Search Strategies for Neural Machine Translationcited by 232
5 more
  1. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  2. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  3. 2023Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Frameworkcited by 16
  4. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 11
  5. 2024Margin-bounded Confidence Scores for Out-of-Distribution Detectioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  2. 2026HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models
  3. 2025PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models
  4. 2024H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models
  5. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  7. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
  8. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models