mlsanityData to 7 Oct 2026

Logit Lens Supervision for Patch-Level Explanations in Vision-Language Models

arXiv 2602.01530 ↗First posted 2 Feb 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Modern autoregressive Vision-Language Models (VLMs) can generate fluent answers while their visual-token representations become weakly tied to the image regions from which they originate. This limits patch-level explainability: a visual token should remain interpretable as the image patch it represents. We study this issue through Logit Lens maps, obtained by projecting each visual-token embedding through the LLM vocabulary head to measure how strongly each image patch is associated with a queried textual concept, such as "cat". We introduce Logit Lens Loss (LLL), a lightweight auxiliary objective that preserves localized visual semantics by directly aligning object-related visual tokens with the vocabulary concepts describing their image regions. LLL requires no architectural modification, mask decoder, or large-scale retraining. We evaluate LLL through both explanation quality and downstream performance. Across LLaVA-v1.5-7B and Qwen2.5-VL-7B, LLL yields sharper object confidence maps, improves grounding and hallucination metrics, transfers to zero-shot pointing, and preserves VQA performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 53 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

48 of 53 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models26 Nov
  2. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  4. 2024F-LMM: Grounding Frozen Large Multimodal Models9 Jun
  5. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  6. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  7. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  8. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  11. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
2 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
22 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2023Vision Transformers Need Registerscited by 844
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  10. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  11. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  12. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  13. 2020Evaluating Weakly Supervised Object Localization Methods Rightcited by 104
  14. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  15. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  16. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  17. 2024Toward Visual Grounding: A Surveycited by 52
  18. 2021Towards Better Explanations of Class Activation Mappingcited by 47
  19. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  20. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  21. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMscited by 17
  22. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Patching to Pruning Visual Computation in Vision Language Models
  2. 2026PatchCue: Enhancing Vision-Language Model Reasoning with Patch-Based Visual Cues
  3. 2023From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
  4. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  5. 2026Language Models Can Explain Visual Features via Steering
  6. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  7. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  8. 2024ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension