mlsanityData to 14 Oct 2026

Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation

arXiv 2404.09127 ↗First posted 14 Apr 2024cs.CLcited by 26

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

Uncertainty estimation is a significant issue for current large language models (LLMs) that are generally poorly calibrated and over-confident, especially with reinforcement learning from human feedback (RLHF). Unlike humans, whose decisions and confidences not only stem from intrinsic beliefs but can also be adjusted through daily observations, existing calibration methods for LLMs focus on estimating or eliciting individual confidence without taking full advantage of the "Collective Wisdom": the interaction among multiple LLMs that can collectively improve both accuracy and calibration. In this work, we propose Collaborative Calibration, a post-hoc training-free calibration strategy that leverages the collaborative and expressive capabilities of multiple tool-augmented LLM agents in a simulated group deliberation process. We demonstrate the effectiveness of Collaborative Calibration on generative QA tasks across various domains, showing its potential in harnessing the rationalization of collectively calibrated confidence assessments and improving the reliability of model predictions.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 42 of its 52 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

42 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration3 Oct
  2. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  3. 2023AdaPlanner: Adaptive Planning from Feedback with Language Models26 May

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
27 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  13. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  14. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  15. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  16. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  17. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  18. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  19. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  20. 2022Prompting GPT-3 To Be Reliablecited by 260
  21. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  22. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  23. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  24. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  25. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  26. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  27. 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 12.

26 papers.

Sort:

By area:

  1. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  2. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  3. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 24
  4. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  5. 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 18
  6. 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  8. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  9. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  10. 2025iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inferencecited by 10
  11. 2025Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Modelscited by 8
  12. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
14 more
  1. 2026Counterfactual Graph for Multi-Agent LLM Calibrationcited by 7
  2. 2026Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoningcited by 5
  3. 2025Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQAcited by 4
  4. 2025BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answeringcited by 3
  5. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  6. 2026Uncertainty Decomposition for Clarification Seeking in LLM Agentscited by 3
  7. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  8. 2025Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Taskscited by 2
  9. 2026Believe Your Model: Distribution-Guided Confidence Calibrationcited by 2
  10. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  11. 2026Confidence Composition for Multiagent Language Model Systemscited by 0
  12. 2026Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debatecited by 0
  13. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  14. 2026From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaborationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs
  2. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
  3. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models
  4. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
  5. 2025Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
  6. 2025Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief
  7. 2026Rethinking Uncertainty Evaluation in Large Language Models
  8. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning