mlsanityData to 7 Oct 2026

Large Language Models Must Be Taught to Know What They Don't Know

arXiv 2406.08391 ↗First posted 12 Jun 2024cs.LG cs.AI cs.CLcited by 99

Not on the map. Why?

Abstract (arXiv)

When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, we show that uncertainty estimates inform human use of LLMs in human-AI collaborative settings through a user study.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 37 of its 67 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

37 of 67 references.

References · 37

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
25 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  13. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  14. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  15. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  16. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  17. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  18. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  19. 2023Do Large Language Models Know What They Don't Know?cited by 238
  20. 2023Fine-tuning Language Models for Factualitycited by 219
  21. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  22. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  23. 2022Uncalibrated Models Can Improve Human-AI Collaborationcited by 26
  24. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  25. 2023Learning Personalized Decision Support Policiescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 37; 2026: 51.

99 papers.

Sort:

By area:

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  2. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  3. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  4. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  5. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  6. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  7. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  8. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 37
  9. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 35
  10. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  11. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  12. 2024A Survey on the Honesty of Large Language Modelscited by 21
38 more
  1. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  2. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  3. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  4. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  5. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  6. 2025Query-Level Uncertainty in Large Language Modelscited by 12
  7. 2026BAGEN: Are LLM Agents Budget-Aware?cited by 12
  8. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
  9. 2026EDIS: Diagnosing LLM Reasoning via Entropy Dynamicscited by 10
  10. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  11. 2025The Geometries of Truth Are Orthogonal Across Taskscited by 9
  12. 2025Do Large Language Models Know What They Are Capable Of?cited by 9
  13. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 8
  14. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  15. 2026AgentAbstain: Do LLM Agents Know When Not to Act?cited by 8
  16. 2026To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Callingcited by 7
  17. 2025AI Awarenesscited by 6
  18. 2026Reaching Beyond the Mode: RL for Distributional Reasoning in Language Modelscited by 6
  19. 2026LLMs Should Express Uncertainty Explicitlycited by 6
  20. 2025None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answeringcited by 5
  21. 2025LLM Agents Are Hypersensitive to Nudgescited by 5
  22. 2025Generalized Correctness Models: Learning Calibrated and Model-Agnostic Correctness Predictors from Historical Patternscited by 5
  23. 2025Can LLMs Refuse Questions They Do Not Know? Measuring Knowledge-Aware Refusal in Factual Taskscited by 5
  24. 2025BrowseConf: Confidence-Guided Test-Time Scaling for Web Agentscited by 5
  25. 2026Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantificationcited by 5
  26. 2026BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidencecited by 5
  27. 2026TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraintscited by 5
  28. 2024Uncertainty-Guided Likelihood Tree Searchcited by 4
  29. 2025Generalization of Fine-Tuned Uncertainty Communication and Metacognition in Large Language Modelscited by 4
  30. 2026Semantic Self-Distillation for Language Model Uncertaintycited by 4
  31. 2026Cross-Model Disagreement as a Label-Free Correctness Signalcited by 4
  32. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 4
  33. 2026Do Coding Agents Understand Least-Privilege Authorization?cited by 4
  34. 2025ConfRAG: Confidence-Guided Retrieval-Augmenting Generationcited by 3
  35. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 3
  36. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
  37. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 3
  38. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3

50 most cited of 99 All 99 (JSON)