# Do Large Language Models Know What They Don't Know?

arXiv: https://arxiv.org/abs/2305.18153 · HTML: https://mlsanity.com/paper/2305.18153/

## Key facts

- Cited by 243 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)

## Built on by (most cited first, 30 of 243)

- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [Why Language Models Hallucinate](https://mlsanity.com/paper/2509.04664/) (2025)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)
- [Large Language Models Must Be Taught to Know What They Don't Know](https://mlsanity.com/paper/2406.08391/) (2024)
- [HalluLens: LLM Hallucination Benchmark](https://mlsanity.com/paper/2504.17550/) (2025)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"](https://mlsanity.com/paper/2410.03727/) (2024)
- [Towards Mitigating Hallucination in Large Language Models via Self-Reflection](https://mlsanity.com/paper/2310.06271/) (2023)
- [LUQ: Long-text Uncertainty Quantification for LLMs](https://mlsanity.com/paper/2403.20279/) (2024)
- [Boosting Language Models Reasoning with Chain-of-Knowledge Prompting](https://mlsanity.com/paper/2306.06427/) (2023)
- [Alignment for Honesty](https://mlsanity.com/paper/2312.07000/) (2023)
- [Sufficient Context: A New Lens on Retrieval Augmented Generation Systems](https://mlsanity.com/paper/2411.06037/) (2024)
- [The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems](https://mlsanity.com/paper/2503.03750/) (2025)
- [Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration](https://mlsanity.com/paper/2310.00280/) (2023)
- [Self-Knowledge Guided Retrieval Augmentation for Large Language Models](https://mlsanity.com/paper/2310.05002/) (2023)
- [Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models](https://mlsanity.com/paper/2506.13206/) (2025)
- [How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study](https://mlsanity.com/paper/2402.16061/) (2024)
- [Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators](https://mlsanity.com/paper/2406.13415/) (2024)
- [When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation](https://mlsanity.com/paper/2402.11457/) (2024)
- [Aligning Language Models to Explicitly Handle Ambiguity](https://mlsanity.com/paper/2404.11972/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks. Current research focuses on enhancing their performance within their existing knowledge. Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend. Therefore, the ability to understand their own limitations on the unknows, referred to as self-knowledge, is of paramount importance. This study aims to evaluate LLMs' self-knowledge by assessing their ability to identify unanswerable or unknowable questions. We introduce an automated methodology to detect uncertainty in the responses of these models, providing a novel measure of their self-knowledge. We further introduce a unique dataset, SelfAware, consisting of unanswerable questions from five diverse categories and their answerable counterparts. Our extensive analysis, involving 20 LLMs including GPT-3, InstructGPT, and LLaMA, discovering an intrinsic capacity for self-knowledge within these models. Moreover, we demonstrate that in-context learning and instruction tuning can further enhance this self-knowledge. Despite this promising insight, our findings also highlight a considerable gap between the capabilities of these models and human proficiency in recognizing the limits of their knowledge.