mlsanityData to 14 Oct 2026

Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

arXiv 2606.10307 ↗First posted 9 Jun 2026cs.CLcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets. We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique. These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 23 of its 26 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

23 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate5 Sep

Other references · 22

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  10. 2020Calibration of Pre-trained Transformerscited by 315
  11. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
10 more
  1. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  2. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  3. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  4. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  5. 2024Detecting Hallucinations in Large Language Model Generation: A Token Probability Approachcited by 42
  6. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  7. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  8. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 19
  9. 2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
  10. 2025Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detectioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Consilience for Verifier-Free Test-Time Scalingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge
  2. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  3. 2025Enhancing Multi-Agent Debate System Performance via Confidence Expression
  4. 2025When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)
  5. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
  6. 2026Understanding and Mitigating Premature Confidence for Better LLM Reasoning
  7. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  8. 2025Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning