mlsanityData to 14 Oct 2026

The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models

arXiv 2609.18453 ↗First posted 16 Sep 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

A calibrated Vision-Language Model (VLM) can repeatedly self-correct, say "Wait, I should recheck," arrive at the wrong answer, and still report high confidence. We find that this occurs because verbalized confidence is largely trajectory-independent in the VLMs and calibration methods we evaluate. We examine this through three complementary lenses: content variation, token masking, and the model's own hesitation markers. We show that confidence is insufficiently sensitive to what the reasoning trajectory actually contains, and that calibration training can paradoxically worsen this disconnect. Since existing metrics like ECE and AUROC cannot detect this problem, we propose the Trajectory-Grounding Score (TGS) in two complementary forms: TGS-self, which compares confidence with and without access to the model's own trajectory, and TGS-pair, which tests whether the model assigns higher confidence to correct trajectories than to flawed ones along the vision, reasoning, and answer axes. We propose TGS-Bench, a model-agnostic suite spanning 10 benchmarks with controlled good/bad trajectory pairs, and show that conventional calibration rankings diverge from trajectory-grounding rankings, exposing a blind spot in current evaluation practice.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 31 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

20 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning10 Apr
  2. 2026Linking Perception, Confidence and Accuracy in MLLMs12 Mar

Other references · 18

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  6. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  7. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  8. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  9. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  10. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  11. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  12. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
6 more
  1. 2024On Verbalized Confidence Scores for LLMscited by 83
  2. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  3. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  4. 2025Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluationcited by 21
  5. 2025SteerConf: Steering LLMs for Confidence Elicitationcited by 20
  6. 2024Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Driftcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models
  2. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation
  3. 2026Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models
  4. 2026Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
  5. 2026Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence
  6. 2025Confidence Calibration in Vision-Language-Action Models
  7. 2025The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models
  8. 2026Answer-Level Trust Selection for Physical Vision-Language Reasoning