# ADVICE: Answer-Dependent Verbalized Confidence Estimation

arXiv: https://arxiv.org/abs/2510.10913 · HTML: https://mlsanity.com/paper/2510.10913/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024)
- [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024)
- [ConfTuner: Training Large Language Models to Express Their Confidence Verbally](https://mlsanity.com/paper/2508.18847/) (2025)
- [SteerConf: Steering LLMs for Confidence Elicitation](https://mlsanity.com/paper/2503.02863/) (2025)
- [Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs](https://mlsanity.com/paper/2506.00582/) (2025)
- [Explainability in Practice: A Survey of Explainable NLP Across Various Domains](https://mlsanity.com/paper/2502.00837/) (2025)
- [Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift](https://mlsanity.com/paper/2409.12428/) (2024)

## Built on by (most cited first, 10 of 10)

- [Closing the Confidence-Faithfulness Gap in Large Language Models](https://mlsanity.com/paper/2603.25052/) (2026)
- [I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering](https://mlsanity.com/paper/2604.03904/) (2026)
- [Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs](https://mlsanity.com/paper/2604.01457/) (2026)
- [Measuring (some aspects of) the metacognition of AI](https://mlsanity.com/paper/2603.29693/) (2026)
- [Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen](https://mlsanity.com/paper/2604.22215/) (2026)
- [Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B](https://mlsanity.com/paper/2604.24070/) (2026)
- [An Analysis of Training-Free Self-Reported Confidence in Language Models](https://mlsanity.com/paper/2609.20541/) (2026)
- [On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models](https://mlsanity.com/paper/2609.32470/) (2026)
- [When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation](https://mlsanity.com/paper/2609.34454/) (2026)
- [Test-time Calibration Learning for Large Language Model Reasoning](https://mlsanity.com/paper/2610.02695/) (2026)

## Abstract (arXiv)

Recent progress in large language models (LLMs) has enabled them to communicate their confidence in natural language, improving transparency and reliability. However, this expressiveness is often accompanied by systematic overconfidence, whose underlying causes remain poorly understood. In this work, we analyze the dynamics of verbalized confidence estimation and identify answer-independence -- the failure to condition confidence on the model's own answer -- as a primary driver of this behavior. To address this, we introduce ADVICE (Answer-Dependent Verbalized Confidence Estimation), a fine-tuning framework that promotes answer-grounded confidence estimation. Extensive experiments show that ADVICE substantially improves confidence calibration, while exhibiting strong generalization to unseen settings without degrading task performance. We further demonstrate that these gains stem from enhanced answer dependence, shedding light on the origins of overconfidence and enabling trustworthy confidence verbalization.