mlsanityData to 7 Oct 2026

DiaVLo: Diagnosing Behaviours of Vision-Language Models

arXiv 2609.22008 ↗First posted 18 Sep 2026cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 98 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

10 of 98 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs24 Feb
  2. 2024Building and better understanding vision-language models: insights and future directions22 Aug

Other references · 8

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  3. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  4. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 104
  5. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  6. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  7. 2025Hidden in plain sight: VLMs overlook their visual representationscited by 10
  8. 2020Learning Post-Hoc Causal Explanations for Recommendationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
  2. 2024VHELM: A Holistic Evaluation of Vision Language Models
  3. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
  4. 2026VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing
  5. 2024What matters when building vision-language models?
  6. 2025LVLM-Aided Alignment of Task-Specific Vision Models
  7. 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding