Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within individual examples. We study answer-level reliability along five-step, question-conditioned evidence-loss trajectories. Using a frozen Qwen2.5-VL-3B-Instruct model, we construct 176 accepted GQA-derived trajectories (880 masking conditions) by progressively masking scene-graph-localized question-critical regions. Native sequence confidence has an evidence monotonicity violation rate (EMVR) of 0.436, and 92.0% of trajectories contain at least one adjacent violation. A matched non-critical-region control shows that full critical masking reduces accuracy by 28.2 percentage points, compared with 0.6 points for equally sized non-critical masks; the paired difference is 27.6 points (95% CI [20.0, 34.7]). We train a lightweight post-hoc reliability head on frozen hidden states, sequence confidence, and entropy. Adding evidence-order supervision to binary cross-entropy (BCE) reduces masking EMVR from 0.330 to 0.303 (paired difference -0.027, 95% CI [-0.044, -0.010]). The same mask-trained objective reduces EMVR from 0.449 to 0.402 on held-out question IDs under unseen local Gaussian blur (difference -0.0468, 95% CI [-0.0739, -0.0199]). AUROC, Brier, and AURC differences between the two learned heads are statistically inconclusive, and native confidence remains stronger for selective-risk ranking. The results separate evidence-order consistency from conventional correctness discrimination rather than establishing generic confidence superiority.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 16 references that are in the map. First posted on arXiv 29 Aug 2026.
Builds on
14 of 16 references.
Closest ancestors · 3
Same direction, newest first
- 2026Evidence-RL: Towards Evidence-intensive Visual Reasoning8 Aug
- 2026Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems16 Apr
- 2025Peek-a-Boo Reasoning: Contrastive Region Masking in MLLMs3 Dec
Other references · 11
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
- 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
- 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
- 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbationcited by 12
- 2026Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Rankingcited by 5
- 2026TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraintcited by 5
- 2024VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Modelscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
- 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
- 2026The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
- 2026SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs
- 2026Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
- 2026Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence
- 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
- 2026DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models