# DiaVLo: Diagnosing Behaviours of Vision-Language Models

arXiv: https://arxiv.org/abs/2609.22008 · HTML: https://mlsanity.com/paper/2609.22008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 98 references that are in the map.
- First posted on arXiv 18 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 10 of 10)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [Question Decomposition Improves the Faithfulness of Model-Generated Reasoning](https://mlsanity.com/paper/2307.11768/) (2023)
- [Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models](https://mlsanity.com/paper/2402.04614/) (2024)
- [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023)
- [No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding](https://mlsanity.com/paper/2503.05061/) (2025)
- [Hidden in plain sight: VLMs overlook their visual representations](https://mlsanity.com/paper/2506.08008/) (2025)
- [Learning Post-Hoc Causal Explanations for Recommendation](https://mlsanity.com/paper/2006.16977/) (2020)
- [Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs](https://mlsanity.com/paper/2602.20878/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.