What Confidence Routing Is Actually Doing: Auditing Routing, Calibration, and Commitment in Multi-Agent Deliberation
Not on the map. Why?
Abstract (arXiv)
A common multi-agent design asks agents to report confidence and lets the highest-scoring agent speak next, implicitly using one scalar both to route the conversation and to estimate uncertainty. We audit this confidence-routed broadcast protocol by separating three trace-level questions: whether it selects the right candidate (routing), whether reported confidence behaves like a probability (calibration), and whether the selected agent publicly states the answer that won the turn (commitment). Our primary study covers 4,181 gpt-oss-120b olympiad-math traces; we repeat the audit on a 2-by-2 actor-by-benchmark grid that adds gemma-4-31B-it and a biology multiple-choice benchmark. In the primary cell, confidence discriminates correct from wrong candidates (AUROC 0.72) but is strongly overconfident (79% mean stated confidence versus 52% accuracy). A cross-fitted, tier-stratified isotonic procedure reduces Expected Calibration Error from 0.278 to 0.008 on held-out candidates, but it does not recover missing discrimination: raw AUROC is only 0.537 and 0.440 in the two Gemma cells. Routing is likewise setting-dependent. Fixed routers differ by at most 1.1 percentage points on gpt-oss/math, whereas raw-confidence argmax performs 5.6 and 11.2 points below random-valid selection in the Gemma cells. Commitment is distinct again: in the primary cell, poll and spoken answers diverge in 20.4% of valid pairs, 62.4% of those revisions are fresh generations, and the unconditional correctness shift is -1.7 points; the other three cells instead range from +0.9 to +12.2 points. The transferable lesson is procedural: routing discrimination, probability calibration, and public commitment must be measured separately before raw confidence is used for deployment decisions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 18 Aug 2026.
Builds on
19 of 32 references.
References · 19
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2018Model Cards for Model Reportingcited by 1,080
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
- 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
- 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
- 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
7 more
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
- 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 114
- 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
- 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 30
- 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoningcited by 29
- 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervisioncited by 9
- 2025DeliberationBench: When Do More Voices Hurt? A Controlled Study of Multi-LLM Deliberation Protocolscited by 2
Built on by
Not cited yet.