mlsanityData to 14 Oct 2026

Calibrating Expressions of Certainty

arXiv 2410.04315 ↗First posted 6 Oct 2024cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 22 of its 38 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

22 of 38 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  11. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  12. 2020Calibration of Pre-trained Transformerscited by 315
10 more
  1. 2019Verified Uncertainty Calibrationcited by 309
  2. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  3. 2015Learning with a Wasserstein Losscited by 196
  4. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  5. 2019Evaluating model calibration in classificationcited by 168
  6. 2020Mitigating Bias in Calibration Error Estimationcited by 101
  7. 2019Distribution Calibration for Regressioncited by 89
  8. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 76
  9. 2016Stabilized Sparse Scaling Algorithms for Entropy Regularized Transport Problemscited by 73
  10. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  2. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  3. 2025Can Large Language Models Express Uncertainty Like Human?cited by 5
  4. 2025MedVAL: Toward Expert-Level Medical Text Validation with Language Modelscited by 3
  5. 2026Retrieval-Augmented Linguistic Calibrationcited by 2
  6. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  7. 2026Inducing Artificial Uncertainty in Language Modelscited by 0