mlsanityData to 7 Oct 2026

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

arXiv 2302.09664 ↗First posted 19 Feb 2023cs.CL cs.AI cs.LGcited by 795

Not on the map. Why?

Cited by 135 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of "semantic equivalence" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.

Key facts. Cited by 795 papers in the map (data through 7 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

31 of 47 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
19 more
  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  2. 2018CoQA: A Conversational Question Answering Challengecited by 674
  3. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  4. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  5. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  6. 2020Calibration of Pre-trained Transformerscited by 307
  7. 2021Unsolved Problems in ML Safetycited by 291
  8. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  9. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  10. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  11. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  12. 2021Entailment as Few-Shot Learnercited by 122
  13. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  14. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  15. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  16. 2020RealFormer: Transformer Likes Residual Attentioncited by 62
  17. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  18. 2020Regression Prior Networkscited by 28
  19. 2014Deep Learning for Answer Sentence Selectioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 50; 2024: 131; 2025: 221; 2026: 393.

795 papers.

Sort:

By area:

  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  4. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  6. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  7. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  8. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  9. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  10. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  11. 2023Fine-tuning Language Models for Factualitycited by 219
  12. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
38 more
  1. 2025Learning to Reason without External Rewardscited by 199
  2. 2023Conformal Language Modelingcited by 163
  3. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  4. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  5. 2024Language Models with Conformal Factuality Guaranteescited by 128
  6. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  7. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  8. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  9. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  10. 2024To Believe or Not to Believe Your LLMcited by 120
  11. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  12. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  13. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  14. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  15. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  16. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  17. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  18. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  19. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  20. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  21. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  22. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  23. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  24. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  25. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  26. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  27. 2024On Verbalized Confidence Scores for LLMscited by 83
  28. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  29. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  30. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  31. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  32. 2024Safeguarding Large Language Models: A Surveycited by 73
  33. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  34. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  35. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  36. 2024Building Guardrails for Large Language Modelscited by 61
  37. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 60
  38. 2024Confidence Regulation Neurons in Language Modelscited by 57

50 most cited of 795 All 795 (JSON)