Data to 7 Oct 2026

Calibration of Pre-trained Transformers

arXiv 2003.07892 ↗First posted 17 Mar 2020cs.CL cs.LGcited by 307

Not on the map. Why?

Abstract (arXiv)

Pre-trained Transformers are now ubiquitous in natural language processing, but despite their high end-task performance, little is known empirically about whether they are calibrated. Specifically, do these models' posterior probabilities provide an accurate empirical measure of how likely the model is to be correct on a given example? We focus on BERT and RoBERTa in this work, and analyze their calibration across three tasks: natural language inference, paraphrase detection, and commonsense reasoning. For each task, we consider in-domain as well as challenging out-of-domain settings, where models face more examples they should be uncertain about. We show that: (1) when used out-of-the-box, pre-trained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as 3.5x lower; (2) temperature scaling is effective at further reducing calibration error in-domain, and using label smoothing to deliberately increase empirical uncertainty helps calibrate posteriors out-of-domain.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 44; 2023: 49; 2024: 68; 2025: 57; 2026: 60.

307 papers.

Sort:

By area:

2022Language Models (Mostly) Know What They Knowcited by 1,783
2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
2022Teaching Models to Express Their Uncertainty in Wordscited by 731
2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
2022Prompting GPT-3 To Be Reliablecited by 258
2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
2023Conformal Language Modelingcited by 163
2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156

38 more

2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
2021Active Learning by Acquiring Contrastive Examplescited by 154
2021On Calibration and Out-of-domain Generalizationcited by 134
2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
2023On the Calibration of Large Language Models and Alignmentcited by 72
2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
2022Stop Measuring Calibration When Humans Disagreecited by 70
2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
2023Selectively Answering Ambiguous Questionscited by 63
2024Calibrating Large Language Models Using Their Generations Onlycited by 63
2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 54
2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
2024Calibration and Correctness of Language Models for Codecited by 49
2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 45
2024Uncertainty Quantification for In-Context Learning of Large Language Modelscited by 45
2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44

50 most cited of 307 All 307 (JSON)