mlsanityData to 7 Oct 2026

Language Models (Mostly) Know What They Know

arXiv 2207.05221 ↗First posted 11 Jul 2022cs.CL cs.AI cs.LGcited by 1,783

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

Key facts. Cited by 1,783 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 32 of its 39 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

32 of 39 references.

References · 32

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
20 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  12. 2020Calibration of Pre-trained Transformerscited by 307
  13. 2022Teaching language models to support answers with verified quotescited by 290
  14. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  15. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  16. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  17. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  18. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  19. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  20. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 178; 2024: 332; 2025: 438; 2026: 812.

1,783 papers.

Sort:

By area:

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
38 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  5. 2023Large Language Modelscited by 774
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  11. 2023Active Retrieval Augmented Generationcited by 506
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  13. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  14. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  15. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  16. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  17. 2023Augmented Language Models: a Surveycited by 398
  18. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  19. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  20. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  21. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  22. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  23. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  24. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  25. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  26. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  27. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  28. 2022Prompting GPT-3 To Be Reliablecited by 258
  29. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  30. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  31. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  32. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  33. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  34. 2023Do Large Language Models Know What They Don't Know?cited by 238
  35. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  36. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  37. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  38. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230

50 most cited of 1,783 All 1,783 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
  2. 2024Confidence in the Reasoning of Large Language Models
  3. 2025The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
  4. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering
  5. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth
  6. 2024Large Language Models Can Self-Correct with Key Condition Verification
  7. 2022An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
  8. 2026A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering