mlsanityData to 14 Oct 2026

Confidence Estimation for LLMs in Multi-turn Interactions

arXiv 2601.02179 ↗First posted 5 Jan 2026cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

While confidence estimation is a promising direction for mitigating hallucinations in Large Language Models (LLMs), current research overwhelmingly focuses on single-turn settings. The dynamics of model confidence in multi-turn conversations, where context accumulates and ambiguity is progressively resolved, remain largely unexplored. This work presents the first systematic study of confidence estimation in multi-turn interactions, establishing a formal evaluation framework grounded in two key desiderata: per-turn calibration and monotonicity of confidence as more information becomes available. To facilitate this, we introduce novel metrics, including a length-normalized Expected Calibration Error (InfoECE), and a new "Hinter-Guesser" paradigm for generating controlled evaluation datasets. Our experiments reveal that widely-used confidence techniques struggle with calibration and monotonicity in multi-turn dialogues. In contrast, a novel logit-based probe we introduce, P(Sufficient), proves comparatively more effective, robustly tracking evidence accumulation and distinguishing it from conversational filler. Our work provides a foundational methodology for developing more reliable and trustworthy conversational agents.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

23 of 28 references.

References · 23

  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  3. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  4. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  5. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  6. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  7. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  8. 2024Can LLM be a Personalized Judge?cited by 100
  9. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  10. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 81
  11. 2023BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialoguescited by 34
  12. 2024Conformity in Large Language Modelscited by 34
11 more
  1. 2024Atomic Calibration of LLMs in Long-Form Generationscited by 24
  2. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 23
  3. 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 21
  4. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  5. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  6. 2024Do We Need Language-Specific Fact-Checking Models? The Case of Chinesecited by 15
  7. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  8. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 10
  9. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  10. 2025GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibrationcited by 5
  11. 2025All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  2. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversitycited by 18
  3. 2026Value of Information: A Framework for Human-Agent Communicationcited by 12
  4. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 6
  5. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  6. 2026When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMscited by 2
  7. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  8. 2026Confidence Should Be Calibrated More Than One Turn Deepcited by 1
  9. 2026Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agentscited by 1
  10. 2026From Sequence to Structure: Relational Uncertainty Propagation for LLM Agentscited by 1
  11. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  12. 2026GRUET: Quantifying Uncertainty of Agentic Reasoning-and-Acting Processescited by 0