mlsanityData to 7 Oct 2026

Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis

arXiv 2210.04714 ↗First posted 10 Oct 2022cs.CL cs.LG stat.MLcited by 104

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) have gained increasing popularity due to their compelling prediction performance in diverse natural language processing (NLP) tasks. When formulating a PLM-based prediction pipeline for NLP tasks, it is also crucial for the pipeline to minimize the calibration error, especially in safety-critical applications. That is, the pipeline should reliably indicate when we can trust its predictions. In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more. Although prior work has looked into some of these considerations, they usually draw conclusions based on a limited scope of empirical studies. There still lacks a holistic analysis on how to compose a well-calibrated PLM-based prediction pipeline. To fill this void, we compare a wide range of popular options for each consideration based on three prevalent NLP classification tasks and the setting of domain shift. In response, we recommend the following: (1) use ELECTRA for PLM encoding, (2) use larger PLMs if possible, (3) use Temp Scaling as the uncertainty quantifier, and (4) use Focal Loss for fine-tuning.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 50 of its 83 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

50 of 83 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  8. 2019When Does Label Smoothing Help?cited by 985
  9. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  10. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  11. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  12. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  13. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  14. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  15. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  16. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  17. 2020Calibration of Pre-trained Transformerscited by 307
  18. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  19. 2021On Calibration and Out-of-domain Generalizationcited by 134
  20. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 128
  21. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  22. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  23. 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
  24. 2020Getting a CLUE: A Method for Explaining Uncertainty Estimatescited by 92
  25. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 77
  26. 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
  27. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  28. 2020Calibrated Reliable Regression using Maximum Mean Discrepancycited by 36
  29. 2020Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functionscited by 34
  30. 2019Pre-trained Language Model for Biomedical Question Answeringcited by 31
  31. 2020Improving Calibration through the Relationship with Adversarial Robustnesscited by 26
  32. 2021SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Miningcited by 24
  33. 2021SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extractioncited by 23
  34. 2020Towards Improving Selective Prediction Ability of NLP Systemscited by 19
  35. 2021Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimatescited by 19
  36. 2020Frequentist Uncertainty in Recurrent Neural Networks via Blockwise Influence Functionscited by 18
  37. 2020Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Datacited by 17
  38. 2020Uncertainty-Aware Semantic Augmentation for Neural Machine Translationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 17; 2024: 48; 2025: 23; 2026: 14.

104 papers.

Sort:

By area:

  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  2. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  3. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  4. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  7. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  8. 2024What Large Language Models Know and What People Think They Knowcited by 109
  9. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  10. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  11. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  12. 2024Safeguarding Large Language Models: A Surveycited by 73
38 more
  1. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  2. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 60
  3. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  4. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 57
  5. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  6. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 51
  7. 2024Uncertainty Quantification for In-Context Learning of Large Language Modelscited by 45
  8. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  9. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  10. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  11. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
  12. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  13. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  14. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  15. 2024Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questionscited by 30
  16. 2023CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agentscited by 28
  17. 2024Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguitycited by 26
  18. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  19. 2024Evaluating language models as risk scorescited by 24
  20. 2024Generative AI in EU Law: Liability, Privacy, Intellectual Property, and Cybersecuritycited by 21
  21. 2024A Survey on the Honesty of Large Language Modelscited by 21
  22. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  23. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  24. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 20
  25. 2023Tree of Uncertain Thoughts Reasoning for Large Language Modelscited by 19
  26. 2025Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibratorcited by 19
  27. 2024Probabilistically Correct Language-based Multi-Robot Planning using Conformal Predictioncited by 18
  28. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  29. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  30. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  31. 2023Conformal Autoregressive Generation: Beam Search with Coverage Guaranteescited by 16
  32. 2026Intelligent AI Delegationcited by 16
  33. 2023Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approachcited by 15
  34. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  35. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 15
  36. 2024Large Language Model Confidence Estimation via Black-Box Accesscited by 14
  37. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  38. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedbackcited by 14

50 most cited of 104 All 104 (JSON)