mlsanityData to 13 Oct 2026

Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

arXiv 2210.15452 ↗First posted 20 Oct 2022cs.CL cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.

Key facts. Cited by 25 papers in the map (data through 13 Oct 2026). Builds on 76 of its 126 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

76 of 126 references.

References · 76

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  5. 2017On Calibration of Modern Neural Networkscited by 4,902
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,430
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,902
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,371
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,278
  11. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 910
  12. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 729
38 more
  1. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 693
  2. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 605
  3. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 529
  5. 2019Quantifying the Carbon Emissions of Machine Learningcited by 519
  6. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 489
  7. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  8. 2019Measuring Calibration in Deep Learningcited by 443
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 440
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 388
  11. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 381
  12. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  13. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  14. 2020Calibration of Pre-trained Transformerscited by 308
  15. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  16. 2020Selective Question Answering under Domain Shiftcited by 231
  17. 2014Recurrent Neural Network Regularizationcited by 189
  18. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  19. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  20. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  21. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
  22. 2020Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIcited by 166
  23. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  24. 2021On Calibration and Out-of-domain Generalizationcited by 134
  25. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 128
  26. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 119
  27. 2019Multilingual is not enough: BERT for Finnishcited by 118
  28. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 101
  29. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  30. 2021Soft Calibration Objectives for Neural Networkscited by 90
  31. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  32. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  33. 2021Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibrationcited by 79
  34. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  35. 2020Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Datacited by 52
  36. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  37. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  38. 2019Energy Usage Reports: Environmental awareness as part of algorithmic accountabilitycited by 44

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 14; 2025: 1; 2026: 7.

25 papers.

Sort:

By area:

  1. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 149
  2. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 69
  3. 2024Calibrating Large Language Models Using Their Generations Onlycited by 64
  4. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  5. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  6. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 57
  7. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  8. 2024Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?cited by 15
  9. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  10. 2024Graph-based Confidence Calibration for Large Language Modelscited by 10
  11. 2024Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domaincited by 9
  12. 2024On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlookcited by 6
13 more
  1. 2024When Quantization Affects Confidence of Large Language Models?cited by 5
  2. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Controlcited by 5
  3. 2024Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation?cited by 3
  4. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  5. 2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
  6. 2024Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weightingcited by 1
  7. 2024Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Taskscited by 1
  8. 2026Quantifying the Uncertainty of Foundation Models with Singular Value Ensemblescited by 1
  9. 2026Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Modelscited by 1
  10. 2024How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?cited by 0
  11. 2026On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluationcited by 0
  12. 2026When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instabilitycited by 0
  13. 2026Conformal Prediction with Paraphrase-Aware Scoring for LLM Uncertainty Quantificationcited by 0