mlsanityData to 7 Oct 2026

HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

arXiv 2604.06165 ↗First posted 7 Apr 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to isolate internal evidence and combines it with a learned prior over external features to recover the true posterior. While intervention-based mitigation methods often degrade utility or fluency by modifying models' internals, we use HaloProbe as an external scoring signal for non-invasive mitigation. Our experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention-based methods while preserving utility.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

30 of 37 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats21 Nov
  2. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs10 Mar
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  4. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations3 Oct
  5. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  6. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models8 May
  7. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
6 more
  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  2. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  3. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  4. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 16
  5. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  6. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models
  2. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
  3. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  4. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token
  5. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  6. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  7. 2026Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
  8. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention