# Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems

arXiv: https://arxiv.org/abs/2510.02377 · HTML: https://mlsanity.com/paper/2510.02377/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Scalable Best-of-N Selection for Large Language Models via Self-Certainty](https://mlsanity.com/paper/2502.18581/) (2025)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)
- [Confidence Improves Self-Consistency in LLMs](https://mlsanity.com/paper/2502.06233/) (2025)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision](https://mlsanity.com/paper/2411.16579/) (2024)
- [Self-Reflection in LLM Agents: Effects on Problem-Solving Performance](https://mlsanity.com/paper/2405.06682/) (2024)
- [Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free](https://mlsanity.com/paper/2410.10814/) (2024)
- [A Study on Large Language Models' Limitations in Multiple-Choice Question Answering](https://mlsanity.com/paper/2401.07955/) (2024)
- [On the Effect of Sampling Diversity in Scaling LLM Inference](https://mlsanity.com/paper/2502.11027/) (2025)
- [EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensembles](https://mlsanity.com/paper/2410.04571/) (2024)

## Built on by (most cited first, 6 of 6)

- [Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities](https://mlsanity.com/paper/2603.10396/) (2026)
- [EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuning](https://mlsanity.com/paper/2601.03725/) (2026)
- [Towards Generation-Efficient Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2605.06053/) (2026)
- [FERA: Uncertainty-Aware Federated Reasoning for Large Language Models](https://mlsanity.com/paper/2605.10082/) (2026)
- [Probing the effectiveness of World Models for Spatial Reasoning through Test-time Scaling](https://mlsanity.com/paper/2512.05809/) (2025)
- [AI-Model Network: Concept, Current State and Future](https://mlsanity.com/paper/2606.27382/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional capabilities, yet selecting the most reliable response from multiple LLMs remains a challenge, particularly in resource-constrained settings. Existing approaches often depend on costly external verifiers, human evaluators, or self-consistency techniques that require multiple samples from a single model. While multi-LLM systems produce more diverse responses than single models and thus have greater potential, they often underperform compared to single LLM self-consistency. We propose a principled, novel and computationally efficient method to select the best response from multiple different LLMs using a calibrated log-likelihood score, implicitly leveraging the inherent knowledge and confidence of these models. Our method demonstrates improvements of approx. 4%, 3%, and 5% across both debate (multi-round LLM discussions) and non-debate (Best-of-N with multiple LLMs) settings on GSM8K, MMLU (6 subsets), and ARC datasets respectively.