GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) hallucinate objects that are not present, and a growing line of work tries to curb this by feeding the model its own generated caption as auxiliary evidence -- assuming that a caption, once available, is something to consume. We show this fails: naively appending a caption can lower accuracy rather than raise it, dropping Qwen2.5-VL-3Bdag on HallusionBench by nearly ten points. To understand why, we build GD-Probe, a diagnostic set that pairs a global and a detail question on the same image, so that any difference in caption effect is attributable to the question alone. Caption utility proves to be a per-query property: the same caption helps global questions and harms detail ones, through a single mechanism -- an embedded caption competes with the image for attention and pulls the model's evidence onto its own text -- whose sign is set by whether the caption covers the queried content. Crucially, this regime is readable from quantities the decoder already emits, with no attention access or grounding. We turn this into GEASS (Gated Evidence-Adaptive Selective Caption Trust), a training-free, logit-level module that decides per query how much of the caption to trust, gating it by the clean path's confidence, weighting it by the entropy reduction it induces, and raising the evidence bar when the two pathways disagree. Across four VLMs and two benchmarks (POPE and HallusionBench), GEASS improves over both vanilla inference and contrastive decoding under a single fixed setting, adding only two forward passes and no parameters.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 39 references that are in the map. First posted on arXiv 3 May 2026.
Builds on
32 of 39 references.
Closest ancestors · 2
Same direction, newest first
- 2025Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning20 Sep
- 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
Other references · 30
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
- 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
18 more
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
- 2018Object Hallucination in Image Captioningcited by 661
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
- 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
- 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
- 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
- 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
- 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
- 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
- 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
- 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
- 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
- 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decodingcited by 19
- 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMscited by 14
- 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decodingcited by 5
- 2025Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Modelscited by 3
- 2025SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answeringcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
- 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
- 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
- 2023Simple Token-Level Confidence Improves Caption Correctness
- 2026PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models
- 2026Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
- 2026Evidence-RL: Towards Evidence-intensive Visual Reasoning
- 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act