MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models
Not on the map. Why?
Abstract (arXiv)
The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluate high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy on LS tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 81 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
46 of 81 references.
References · 46
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
34 more
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
- 2017Crowdsourcing Multiple Choice Science Questionscited by 742
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
- 2019How multilingual is Multilingual BERT?cited by 642
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
- 2023A Survey of Hallucination in Large Foundation Modelscited by 365
- 2020Calibration of Pre-trained Transformerscited by 307
- 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
- 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
- 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
- 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
- 2018Correcting Length Bias in Neural Machine Translationcited by 119
- 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
- 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
- 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
- 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
- 2024Calibrating Large Language Models with Sample Consistencycited by 72
- 2020Inducing Language-Agnostic Multilingual Representationscited by 48
- 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
- 2023Multilingual Previously Fact-Checked Claim Retrievalcited by 32
- 2024Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questionscited by 30
- 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
- 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
- 2021Bayesian Transformer Language Models for Speech Recognitioncited by 22
- 2023Enhancing Large Language Models Against Inductive Instructions with Dual-critique Promptingcited by 21
- 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
- 2024Linguini: A benchmark for language-agnostic linguistic reasoningcited by 17
- 2023Towards Unifying Multi-Lingual and Cross-Lingual Summarizationcited by 15
- 2022Bayesian Neural Network Language Modeling for Speech Recognitioncited by 3
Built on by
Not cited yet.