mlsanityData to 14 Oct 2026

Calibration of Natural Language Understanding Models with Venn--ABERS Predictors

arXiv 2205.10586 ↗First posted 21 May 2022cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn--ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval -- all while retaining the original model's predictive accuracy.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 17 of its 35 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

17 of 35 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  11. 2020Calibration of Pre-trained Transformerscited by 315
  12. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
5 more
  1. 2013Learning by Transductioncited by 121
  2. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 76
  3. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  4. 2021Conformal prediction for text infilling and part-of-speech predictioncited by 22
  5. 2015Large-scale probabilistic predictors with and without guarantees of validitycited by 18

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5.

6 papers.

  1. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 53
  2. 2024Conformal Prediction: A Data Perspectivecited by 25
  3. 2023Evaluating Machine Translation Quality with Conformal Predictive Distributionscited by 9
  4. 2024Conformalized Link Prediction on Graph Neural Networkscited by 9
  5. 2024Calibrated Large Language Models for Binary Question Answeringcited by 3
  6. 2024A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreementcited by 3