mlsanityData to 7 Oct 2026

Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens

arXiv 2509.03025 ↗First posted 3 Sep 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs. However, our finding reveals they often mistakenly perceive text inputs lacking visual evidence as being part of the image, leading to erroneous responses. In light of this finding, we probe whether LVLMs possess an internal capability to determine if textual concepts are grounded in the image, and discover a specific subset of Feed-Forward Network (FFN) neurons, termed Visual Absence-aware (VA) neurons, that consistently signal the visual absence through a distinctive activation pattern. Leveraging these patterns, we develop a detection module that systematically classifies whether an input token is visually grounded. Guided by its prediction, we propose a method to refine the outputs by reinterpreting question prompts or replacing the detected absent tokens during generation. Extensive experiments show that our method effectively mitigates the models' tendency to falsely presume the visual presence of text input and its generality across various LVLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 30 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

20 of 30 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  10. 2016Language Modeling with Gated Convolutional Networkscited by 688
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2018Object Hallucination in Image Captioningcited by 661
8 more
  1. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  2. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  5. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  6. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  7. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  8. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Neuron-Guided Fine-Tuning: Unlocking Efficient Alignment Mechanisms for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Intermediate States: Explaining Visual Redundancy through Language
  2. 2026Diagnosing Visual Ignorance in Vision-Language Models
  3. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  4. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow
  5. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  6. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  7. 2026Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
  8. 2025Where do Large Vision-Language Models Look at when Answering Questions?