Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Label-free reliability for vision-language models rests on invariance: perturb the input and a faithful reader's answer should not change. This has a known blind spot, a systematic misreading survives the perturbation and gets certified wrong, which we show is computable, not just real: an error is invisible to an edit exactly when the two commute, so the errors a suite cannot reach form its joint centralizer, a set that shrinks as edits are added and can be written down rather than guessed at. We act on the complementary relation, equivariance: edit a figure's data and the correct answer must change by a computable amount. Two matched edits are provably complete for affine reading errors; no suite of swap edits is complete for label permutations, and cyclic relabeling closes most of that gap. We instantiate the theory as the Equivariance-Consistency Score, a label-free, training-free detector, and release REND-EQUIV, pairing matched invariance and equivariance sets over identical data. The predicted ordering holds across three models and a hand-labeled population immune to the one circularity in how it is selected; a second invariance-family method confirms the blind spot belongs to the relation, not to any implementation; and cyclic relabeling delivers its predicted gain on a matched real sample. The same characterization explains a reported inversion of this ordering in the classifier metamorphic-testing literature: detectability is a joint property of the relation and the fault class, never of the relation alone.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 32 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
25 of 32 references.
Closest ancestors · 5
Same direction, newest first
- 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems19 May
- 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May
- 2024Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison10 Jul
- 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering16 Apr
- 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning23 Feb
Other references · 20
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
- 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
- 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
- 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
- 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoningcited by 8
8 more
- 2025Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucinationcited by 7
- 2025Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoningcited by 7
- 2025ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understandingcited by 5
- 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
- 2026Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelinescited by 4
- 2026Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Surveycited by 3
- 2025Chart-CoCa: Self-Improving Chart Understanding of Vision LMs via Code-Driven Synthesis and Candidate-Conditioned Answeringcited by 2
- 2026Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understandingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Test-Time Consistency in Vision Language Models
- 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models
- 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Models
- 2026The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models
- 2026Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
- 2026Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models
- 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
- 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding