# Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence

arXiv: https://arxiv.org/abs/2609.09184 · HTML: https://mlsanity.com/paper/2609.09184/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 29 Aug 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering](https://mlsanity.com/paper/1902.09506/) (2019)
- [Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation](https://mlsanity.com/paper/2504.14848/) (2025)
- [Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems](https://mlsanity.com/paper/2604.14799/) (2026)
- [Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking](https://mlsanity.com/paper/2605.10893/) (2026)
- [TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint](https://mlsanity.com/paper/2608.13167/) (2026)
- [VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2412.15739/) (2024)
- [Evidence-RL: Towards Evidence-intensive Visual Reasoning](https://mlsanity.com/paper/2608.08021/) (2026)
- [Peek-a-Boo Reasoning: Contrastive Region Masking in MLLMs](https://mlsanity.com/paper/2512.08976/) (2025)

## Abstract (arXiv)

Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within individual examples. We study answer-level reliability along five-step, question-conditioned evidence-loss trajectories. Using a frozen Qwen2.5-VL-3B-Instruct model, we construct 176 accepted GQA-derived trajectories (880 masking conditions) by progressively masking scene-graph-localized question-critical regions. Native sequence confidence has an evidence monotonicity violation rate (EMVR) of 0.436, and 92.0% of trajectories contain at least one adjacent violation. A matched non-critical-region control shows that full critical masking reduces accuracy by 28.2 percentage points, compared with 0.6 points for equally sized non-critical masks; the paired difference is 27.6 points (95% CI [20.0, 34.7]). We train a lightweight post-hoc reliability head on frozen hidden states, sequence confidence, and entropy. Adding evidence-order supervision to binary cross-entropy (BCE) reduces masking EMVR from 0.330 to 0.303 (paired difference -0.027, 95% CI [-0.044, -0.010]). The same mask-trained objective reduces EMVR from 0.449 to 0.402 on held-out question IDs under unseen local Gaussian blur (difference -0.0468, 95% CI [-0.0739, -0.0199]). AUROC, Brier, and AURC differences between the two learned heads are statistically inconclusive, and native confidence remains stronger for selective-risk ranking. The results separate evidence-order consistency from conventional correctness discrimination rather than establishing generic confidence superiority.