# Questioning the Stability of Visual Question Answering

arXiv: https://arxiv.org/abs/2511.11206 · HTML: https://mlsanity.com/paper/2511.11206/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 14 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Built on by (most cited first, 5 of 5)

- [TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering](https://mlsanity.com/paper/2603.09696/) (2026)
- [Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations](https://mlsanity.com/paper/2608.04885/) (2026)
- [When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering](https://mlsanity.com/paper/2608.05670/) (2026)
- [Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning](https://mlsanity.com/paper/2609.28851/) (2026)
- [On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs](https://mlsanity.com/paper/2610.19892/) (2026)

## Abstract (arXiv)

Visual Language Models (VLMs) have achieved remarkable progress, yet their reliability under small, meaning-preserving input changes remains poorly understood. We present the first large-scale, systematic study of VLM robustness to benign visual and textual perturbations: pixel-level shifts, light geometric transformations, padded rescaling, paraphrasing, and multilingual rewrites that do not alter the underlying semantics of an image-question pair. Across a broad set of models and datasets, we find that modern VLMs are highly sensitive to such minor perturbations: a substantial fraction of samples change their predicted answer under at least one visual or textual modification. We characterize how this instability varies across perturbation types, question categories, and models, revealing that even state-of-the-art systems (e.g., GPT-4o, Gemini 2.0 Flash) frequently fail under shifts as small as a few pixels or harmless rephrasings. We further show that sample-level stability serves as a strong indicator of correctness: stable samples are consistently far more likely to be answered correctly. Leveraging this, we demonstrate that the stability patterns of small, accessible open-source models can be used to predict the correctness of much larger closed-source models with high precision. Our findings expose a fundamental fragility in current VLMs and highlight the need for robustness evaluations that go beyond adversarial perturbations, focusing instead on invariances that models should reliably uphold.