mlsanityData to 14 Oct 2026

Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null

arXiv 2610.00024 ↗First posted 1 Sep 2026cs.CV cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Across three vision-language model architectures (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3-8B), we report a universal negative finding for mid-layer interpretability. On POPE -- the benchmark common to all three -- the mid layers encode the ground-truth answer in 68-91% of errors, yet this signal is not causally active for the final prediction: residual-stream patching yields 0% non-trivial flip at the layer level on all three architectures, and on two of three at the per-head level (Qwen: 0/12,600 patched forwards). The lone exception, InternVL3 layer-20 head-2, is a non-vocab, self-attending head whose effect is localized to that specific head (p < 1e-4). Despite the null, the errors separate operationally into three failure modes -- Perception Failure, Encoded-but-Disconnected, Prior-Override -- learnable above 60% on all three architectures, and the architecture's prior direction predicts which of two interventions elicits a category-specific response. We report these mitigation effects under oracle labels as evidence the categories are mechanistically real, not as a deployable method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

12 of 17 references.

References · 12

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  8. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  9. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  10. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  11. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similaritiescited by 7
  12. 2026To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMscited by 3

Built on by

Not cited yet.