It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Test-time scaling lifts large language model reasoning by sampling many candidate solutions and selecting among them, yet the same recipe transfers poorly to vision-language models (VLMs): recent work shows that simple majority voting beats selection methods built on the model's own self-verification, apparently because at the selection layer an image-grounded answer and a confident guess from the language prior look the same. A natural fix is to make the selection signal one that cannot be computed without the image. We study Perturbation Grounded Selection (Pgs), a label-free, training-free rule that scores each candidate by whether the model re-derives it under label-preserving perturbations of the input (cropping, background masking, mild photometric or geometric jitter); Pgs recovers majority voting when the perturbation set is empty. The decisive question is not whether Pgs beats chain-of-thought only majority voting, but whether the perturbation term adds anything once decoding format and budget are controlled. We therefore introduce a format-matched control (MatchedCtrl): the same short, no-CoT draws spent on the original image. Across TextVQA, MATH-Vision, MMMU, and ViLP, with a Qwen headline (three-seed means) and LLaVA-OneVision coverage in matched-budget selector tables, Pgs appears to beat plain majority voting by up to +31.8 points on TextVQA (Qwen), but MatchedCtrl tracks or exceeds Pgs within noise on every benchmark, including the vision-required ViLP; no Qwen category shows a significant gain over this control. The stability gap is real and image-dependent (up to +0.48), yet does not predict per-instance wins. The result is negative and diagnostic: perturbation consistency is at best a partial diagnostic of visual dependence and, on its own, not a usable selection signal once format is controlled; gains reported against CoT-only majority voting overstate such methods.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 35 references that are in the map. First posted on arXiv 2 Aug 2026.
Builds on
34 of 35 references.
Closest ancestors · 13
Same direction, newest first
- 2026Test-Time Scaling for Small VLMs on Multilingual Visual MCQ10 Jul
- 2026On Test-Time Scaling for Vision-Language Models27 Jun
- 2026AVIS: Adaptive Test-Time Scaling for Vision-Language Models10 Jun
- 2026Answer Self-Consistency with Margin-Triggered Question Re-Arbitration for the CVPR 2026 VidLLMs Challenge3 Jun
- 2026Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models29 May
- 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs6 Feb
- 2025Limits and Gains of Test-Time Scaling in Vision-Language Reasoning11 Dec
- 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers9 Dec
- 2025Efficient Test-Time Scaling for Small Vision-Language Models3 Oct
- 2025Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?20 Jun
- 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use25 May
- 2024Probing Visual Language Priors in VLMs31 Dec
Other references · 21
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2019Towards VQA Models That Can Readcited by 1,696
- 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
- 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
- 2025Confidence Improves Self-Consistency in LLMscited by 111
- 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
9 more
- 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
- 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
- 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
- 2025Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoningcited by 8
- 2025Test-Time Consistency in Vision Language Modelscited by 5
- 2026Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelinescited by 4
- 2026Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulatorscited by 4
- 2026Same Answer, Different Representations: Hidden instability in VLMscited by 2
- 2026ARBITER: Reasoning Trajectory Basins and Majority Vote Failures in Test-Time Samplingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
- 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering
- 2026Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning
- 2026Answer-Level Trust Selection for Physical Vision-Language Reasoning
- 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
- 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
- 2025Token-Level Inference-Time Alignment for Vision-Language Models
- 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models