mlsanityData to 7 Oct 2026

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

arXiv 2603.16728 ↗First posted 17 Mar 2026cs.LGcited by 5

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) are increasingly deployed in high-stakes settings where reliable uncertainty quantification (UQ) is as important as predictive accuracy. Extended reasoning via chain-of-thought (CoT) prompting or reasoning-trained models has become ubiquitous in modern VLM pipelines, yet its effect on UQ reliability remains poorly understood. Our results show that reasoning tends to degrade the quality of many uncertainty estimates, even when it improves task accuracy. We identify implicit answer conditioning as the primary mechanism: as reasoning traces converge on a conclusion before the final answer is generated, token probabilities increasingly reflect consistency with the model's own reasoning trace rather than uncertainty about correctness. In effect, the model becomes overconfident in its answer. In contrast, agreement-based consistency remains robust and often improves under reasoning, making it a practical choice for uncertainty estimation in reasoning-enabled VLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 35 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

28 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning23 Feb
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 26

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2025Qwen3-VL Technical Reportcited by 2,534
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  11. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
14 more
  1. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  2. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  3. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 106
  4. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  5. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  6. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  7. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  8. 2023Selectively Answering Ambiguous Questionscited by 63
  9. 2025Reasoning Models Better Express Their Confidencecited by 63
  10. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  11. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 31
  12. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  13. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 28
  14. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Driftcited by 1
  2. 2026Passive Construction Site Safety Monitoring via Persona-Scaffolded Adversarial Chain-of-Thought VLM Verificationcited by 0
  3. 2026When Thinking Hurts: Epistemic Signals in the Reasoning Chains of Visual Language Modelscited by 0
  4. 2026EVADE: Evidence-Verified Agentic Diagnosis with Escapecited by 0
  5. 2026Rethinking Probability-Based Reinforcement Learning From Posterior Concentrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Calibration Drift Under Reasoning: How Chain-of-Thought Budgets Induce Overconfidence in Large Language Models
  2. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?
  3. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models
  4. 2025The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models
  5. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
  6. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
  7. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
  8. 2026Amplified Does Not Mean Predictive: Reasoning Behaviors in Thinking Models