mlsanityData to 7 Oct 2026

MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

arXiv 2402.13606 ↗First posted 21 Feb 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluated high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy in LS scenarios.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 81 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

41 of 81 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
29 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  5. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2019How multilingual is Multilingual BERT?cited by 642
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  12. 2020Calibration of Pre-trained Transformerscited by 307
  13. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  14. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  15. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  16. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  17. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  18. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  19. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  20. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  21. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  22. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  23. 2023Multilingual Previously Fact-Checked Claim Retrievalcited by 32
  24. 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
  25. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  26. 2021Bayesian Transformer Language Models for Speech Recognitioncited by 22
  27. 2023Enhancing Large Language Models Against Inductive Instructions with Dual-critique Promptingcited by 21
  28. 2023Towards Unifying Multi-Lingual and Cross-Lingual Summarizationcited by 15
  29. 2022Bayesian Neural Network Language Modeling for Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 9
  2. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Modelscited by 7
  3. 2025Self-Error-Instruct: Generalizing from Errors for LLMs Mathematical Reasoningcited by 2
  4. 2026The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructionscited by 1