mlsanityData to 14 Oct 2026

PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models

arXiv 2608.21819 ↗First posted 22 Aug 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Reliable image captioning in Vision-Language Models (VLMs) requires captions to be both precise and complete, avoiding unsupported object mentions while covering visible objects. Existing training-free methods primarily address the former requirement, suppressing unsupported object words by intervening on model-predicted mentions during generation. Because they operate only on objects the model is already likely to mention, visible objects omitted from the output remain difficult to recover. We propose PatchGate, a training-free framework that extracts prompt-free object evidence intrinsic to a frozen VLM before generation and uses it to narrow the gap between an intrinsic object set and final object mentions. In the first stage, Visual Evidence eXtraction (VEX) reads patch-level lexical evidence from the latter half of LM decoder layers and constructs an image-conditioned object set without any task prompt. In the second stage, Visual-Evidence Inclusion-Exclusion Decoding (VIED) uses this object evidence to calibrate decoding logits, promoting evidence-supported but under-verbalized objects and suppressing weakly supported but over-verbalized objects. On AMBER, PatchGate improves both sides of object-level reliability, increasing visible-object coverage from 49.4 to 56.0 (+13.4%) and reducing object hallucination by lowering CHAIR from 7.5 to 6.6 (-12.0%), without external detectors or fine-tuning and with one extra forward pass.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 36 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

33 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct

Other references · 32

  1. 2020End-to-End Object Detection with Transformerscited by 7,414
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019Root Mean Square Layer Normalizationcited by 1,480
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  12. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
20 more
  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  4. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  5. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  6. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 94
  7. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  8. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  9. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  10. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  11. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  12. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 65
  13. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46
  14. 2024Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMscited by 22
  15. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 14
  16. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMscited by 8
  17. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  18. 2024TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Biascited by 4
  19. 2026Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancycited by 2
  20. 2026Breaking the Illusion: When Positive Meets Negative in Multimodal Decodingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Logit Lens Supervision for Patch-Level Explanations in Vision-Language Models
  2. 2026DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models
  3. 2023Simple Token-Level Confidence Improves Caption Correctness
  4. 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
  5. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  6. 2025Context Matters: Learning Global Semantics via Object-Centric Representation
  7. 2026From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models
  8. 2026Improving Reasoning in Vision-Language Models via Perception Verified Self-Training