mlsanityData to 14 Oct 2026

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

arXiv 2607.27823 ↗First posted 30 Jul 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6% while retaining 99.6% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIRi/CHAIRs by 37.0%/30.4% without shortening captions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 33 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

22 of 33 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token5 Mar
  2. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification14 Feb
  3. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering8 Jan
  4. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity27 Aug
  5. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow20 May
  6. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling17 Apr
  7. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  8. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  9. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  10. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  12. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
1 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 9

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2018Object Hallucination in Image Captioningcited by 669
  8. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  9. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  2. 2026VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs
  3. 2026ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models
  4. 2026Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
  5. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  6. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors
  7. 2026RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction
  8. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation