mlsanityData to 14 Oct 2026

Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

arXiv 2604.17112 ↗First posted 18 Apr 2026cs.AIcited by 10

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution to more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed as the gap between inter-model and intra-model sequence-semantic similarity. We then define total uncertainty (TU) as the sum of AU and EU. In a comprehensive study across five 7-9B instruction-tuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 55 of its 68 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

55 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute1 Apr
  2. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers27 Feb
  3. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certainty25 Feb
  4. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 51

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2018CoQA: A Conversational Question Answering Challengecited by 683
  11. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  12. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  13. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  14. 2024Measuring short-form factuality in large language modelscited by 373
  15. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  16. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  17. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  18. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  19. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  20. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  21. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  22. 2024To Believe or Not to Believe Your LLMcited by 120
  23. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  24. 2024The rising costs of training frontier AI modelscited by 95
  25. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 76
  26. 2023LoRA ensembles for large language model fine-tuningcited by 67
  27. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 62
  28. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  29. 2024SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Modelscited by 56
  30. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  31. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  32. 2023Introducing an Improved Information-Theoretic Measure of Predictive Uncertaintycited by 28
  33. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  34. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 19
  35. 2025Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detectioncited by 18
  36. 2024(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Modelscited by 15
  37. 2025Enhancing Hallucination Detection through Noise Injectioncited by 13
  38. 2025Uncertainty-Aware Fusion: An Ensemble Framework for Mitigating Hallucinations in Large Language Modelscited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agentscited by 4
  2. 2026Credal Large Language Models for Semantic Commitment under Uncertaintycited by 2
  3. 2026Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMscited by 1
  4. 2026Sixteen models, fewer than two voices: measuring ensemble dispersion where no answer is uniquely correctcited by 0
  5. 2026Decomposing Wrong-Consensus Agreement in LLM Self-Consistencycited by 0
  6. 2026From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMscited by 0
  7. 2026Target-Checked Reliability Score Refinement for Video Question Answeringcited by 0
  8. 2026Observational Equivalence of LLM and Human Annotationcited by 0
  9. 2026Pinocchio: Fast Uncertainty Estimates for Black-Box Language Modelscited by 0
  10. 2026On the Limits of Metacognitive Monitoring in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Semantic Uncertainty Quantification Needs Factual Equivalence
  2. 2026ChainUQ: Reasoning Consistency-Aware Uncertainty Quantification for Large Language Models
  3. 2024Distinguishing the Knowable from the Unknowable with Language Models
  4. 2026Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scores
  5. 2025Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal
  6. 2023Quantifying Uncertainty in Natural Language Explanations of Large Language Models
  7. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
  8. 2026Rethinking Uncertainty Evaluation in Large Language Models