mlsanityData to 14 Oct 2026

Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems

arXiv 2510.02377 ↗First posted 30 Sep 2025cs.CL cs.LGcited by 6

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional capabilities, yet selecting the most reliable response from multiple LLMs remains a challenge, particularly in resource-constrained settings. Existing approaches often depend on costly external verifiers, human evaluators, or self-consistency techniques that require multiple samples from a single model. While multi-LLM systems produce more diverse responses than single models and thus have greater potential, they often underperform compared to single LLM self-consistency. We propose a principled, novel and computationally efficient method to select the best response from multiple different LLMs using a calibrated log-likelihood score, implicitly leveraging the inherent knowledge and confidence of these models. Our method demonstrates improvements of approx. 4%, 3%, and 5% across both debate (multi-round LLM discussions) and non-debate (Best-of-N with multiple LLMs) settings on GSM8K, MMLU (6 subsets), and ARC datasets respectively.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 18 of its 21 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

18 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debate23 May

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  8. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  9. 2024To Believe or Not to Believe Your LLMcited by 120
  10. 2025Confidence Improves Self-Consistency in LLMscited by 113
  11. 2024Efficient multi-prompt evaluation of LLMscited by 73
  12. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
5 more
  1. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 39
  2. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  3. 2024A Study on Large Language Models' Limitations in Multiple-Choice Question Answeringcited by 22
  4. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  5. 2024EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 5
  2. 2026EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuningcited by 2
  3. 2026Towards Generation-Efficient Uncertainty Estimation in Large Language Modelscited by 2
  4. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  5. 2025Probing the effectiveness of World Models for Spatial Reasoning through Test-time Scalingcited by 0
  6. 2026AI-Model Network: Concept, Current State and Futurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Refining Answer Distributions for Improved Large Language Model Reasoning
  2. 2025Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools
  3. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth
  4. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
  5. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  6. 2026Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
  7. 2025CER: Confidence Enhanced Reasoning in LLMs
  8. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty