# Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty

arXiv: https://arxiv.org/abs/2412.09572 · HTML: https://mlsanity.com/paper/2412.09572/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning.
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Placed in

- LLM reasoning / [Hallucination detection and mitigation](https://mlsanity.com/lr/hallucination-detection-and-mitigation/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [To Believe or Not to Believe Your LLM](https://mlsanity.com/paper/2406.02543/) (2024)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [HALO: An Ontology for Representing and Categorizing Hallucinations in Large Language Models](https://mlsanity.com/paper/2312.05209/) (2023)

## Built on by (most cited first, 11 of 11)

- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [Position: Towards a Responsible LLM-empowered Multi-Agent Systems](https://mlsanity.com/paper/2502.01714/) (2025)
- [Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems](https://mlsanity.com/paper/2506.10060/) (2025)
- [Cross-Model Disagreement as a Label-Free Correctness Signal](https://mlsanity.com/paper/2603.25450/) (2026)
- [Counterfactual Graph for Multi-Agent LLM Calibration](https://mlsanity.com/paper/2605.30653/) (2026)
- [Agentic LLM Framework for Adaptive Decision Discourse](https://mlsanity.com/paper/2502.10978/) (2025)
- [ConfAgents: A Conformal-Guided Multi-Agent Framework for Cost-Efficient Medical Diagnosis](https://mlsanity.com/paper/2508.04915/) (2025)
- [Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier](https://mlsanity.com/paper/2606.20662/) (2026)
- [DanceHA: A Multi-Agent Framework for Document-Level Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2603.16546/) (2026)
- [PopPy: Opportunistically Exploiting Parallelism in Python Agentic Workflows](https://mlsanity.com/paper/2605.18697/) (2026)

## Abstract (arXiv)

Quantifying uncertainty in black-box LLMs is vital for reliable responses and scalable oversight. Existing methods, which gauge a model's uncertainty through evaluating self-consistency in responses to the target query, can be misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same target query when answering a knowledge-preserving perturbation of the query. We systematically analyze the model behaviors and demonstrate that this discrepancy stems from suboptimal retrieval of parametric knowledge, often due to contextual biases that prevent consistent access to stored knowledge. We then introduce DiverseAgentEntropy, a novel, theoretically-grounded method employing multi-agent interaction across diverse query variations for uncertainty estimation of black-box LLMs. This approach more accurately assesses an LLM's true uncertainty and improves hallucination detection, outperforming existing self-consistency based techniques.