mlsanityData to 14 Oct 2026

ChatGPT Prompting Cannot Estimate Predictive Uncertainty in High-Resource Languages

arXiv 2311.06427 ↗First posted 10 Nov 2023cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

ChatGPT took the world by storm for its impressive abilities. Due to its release without documentation, scientists immediately attempted to identify its limits, mainly through its performance in natural language processing (NLP) tasks. This paper aims to join the growing literature regarding ChatGPT's abilities by focusing on its performance in high-resource languages and on its capacity to predict its answers' accuracy by giving a confidence level. The analysis of high-resource languages is of interest as studies have shown that low-resource languages perform worse than English in NLP tasks, but no study so far has analysed whether high-resource languages perform as well as English. The analysis of ChatGPT's confidence calibration has not been carried out before either and is critical to learn about ChatGPT's trustworthiness. In order to study these two aspects, five high-resource languages and two NLP tasks were chosen. ChatGPT was asked to perform both tasks in the five languages and to give a numerical confidence value for each answer. The results show that all the selected high-resource languages perform similarly and that ChatGPT does not have a good confidence calibration, often being overconfident and never giving low confidence values.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 23 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

13 of 23 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  7. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  8. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  9. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  10. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  11. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  12. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 65
1 more
  1. 2023Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processingcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025One For All: LLM-based Heterogeneous Mission Planning in Precision Agriculturecited by 2
  2. 2025Solomonoff-Inspired Hypothesis Ranking with LLMs for Prediction Under Uncertaintycited by 0