mlsanityData to 7 Oct 2026

Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

arXiv 2607.06327 ↗First posted 7 Jul 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 17 of its 23 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

17 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certainty25 Feb
  2. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  5. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  6. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  7. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  8. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  9. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  10. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  11. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  12. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 73
3 more
  1. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
  2. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  3. 2025Monte Carlo Temperature: a robust sampling strategy for LLM's uncertainty quantification methodscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions
  2. 2025OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Data
  3. 2026Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
  4. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study
  5. 2025Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMs
  6. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  7. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty
  8. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey