mlsanityData to 14 Oct 2026

Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method

arXiv 2310.17918 ↗First posted 27 Oct 2023cs.CL cs.AIcited by 51

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown great potential in Natural Language Processing (NLP) tasks. However, recent literature reveals that LLMs generate nonfactual responses intermittently, which impedes the LLMs' reliability for further utilization. In this paper, we propose a novel self-detection method to detect which questions that a LLM does not know that are prone to generate nonfactual results. Specifically, we first diversify the textual expressions for a given question and collect the corresponding answers. Then we examine the divergencies between the generated answers to identify the questions that the model may generate falsehoods. All of the above steps can be accomplished by prompting the LLMs themselves without referring to any other external resources. We conduct comprehensive experiments and demonstrate the effectiveness of our method on recently released LLMs, e.g., Vicuna, ChatGPT, and GPT-4.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

52 of 72 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2017On Calibration of Modern Neural Networkscited by 4,960
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  13. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  14. 2020Recipes for building an open-domain chatbotcited by 593
  15. 2016Deep Exploration via Bootstrapped DQNcited by 527
  16. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  17. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  18. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  19. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  20. 2020Calibration of Pre-trained Transformerscited by 315
  21. 2021Unsolved Problems in ML Safetycited by 292
  22. 2022Generating Sequences by Learning to Self-Correctcited by 277
  23. 2022Prompting GPT-3 To Be Reliablecited by 260
  24. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  25. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
  26. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  27. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  28. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  29. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  30. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  31. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  32. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  33. 2023Can ChatGPT Assess Human Personalities? A General Evaluation Frameworkcited by 70
  34. 2021FaVIQ: FAct Verification from Information-seeking Questionscited by 36
  35. 2018ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusterscited by 29
  36. 2020Improving Calibration through the Relationship with Adversarial Robustnesscited by 26
  37. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25
  38. 2023UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Predictioncited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 21; 2025: 17; 2026: 10.

51 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  3. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  5. 2024To Believe or Not to Believe Your LLMcited by 120
  6. 2024Can AI Assistants Know What They Don't Know?cited by 50
  7. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 41
  8. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  9. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 27
  10. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  11. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  12. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
38 more
  1. 2023Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankerscited by 19
  2. 2025Reconsidering LLM Uncertainty Estimation Methods in the Wildcited by 17
  3. 2024DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Modelscited by 15
  4. 2024Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Stylecited by 11
  5. 2025Do Large Language Models Know How Much They Know?cited by 11
  6. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 11
  7. 2025Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulnesscited by 10
  8. 2024To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguitycited by 8
  9. 2025Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detectioncited by 8
  10. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  11. 2024Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answeringcited by 7
  12. 2024PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Modelscited by 6
  13. 2025AutoMathKG: The automated mathematical knowledge graph based on LLM and vector databasecited by 6
  14. 2026Where Knowledge Collides: A Mechanistic Study of Intra-Memory Knowledge Conflict in Language Modelscited by 5
  15. 2024Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completioncited by 4
  16. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  17. 2025TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputscited by 4
  18. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Modelscited by 4
  19. 2024Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editingcited by 3
  20. 2024VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMscited by 3
  21. 2025Exploring How LLMs Capture and Represent Domain-Specific Knowledgecited by 3
  22. 2024Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responsescited by 2
  23. 2025Large Language Models Could Be Rote Learnerscited by 2
  24. 2025Conflicts in Texts: Data, Implications and Challengescited by 2
  25. 2025Know Or Not: a library for evaluating out-of-knowledge base robustnesscited by 2
  26. 2025When Facts Change: Probing LLMs on Evolving Knowledge with evolveQAcited by 2
  27. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  28. 2025Resolving Knowledge Conflicts in Domain-specific Data Selection: A Case Study on Medical Instruction-tuningcited by 1
  29. 2025HalluClean: A Unified Framework to Combat Hallucinations in LLMscited by 1
  30. 2026Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"cited by 1
  31. 2026REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignmentcited by 1
  32. 2026HIVE: Hidden-Evidence Verification for Hallucination Detection in Diffusion Large Language Modelscited by 1
  33. 2026Prompt-Based Abstention Fails Under Misleading Context: A Controlled Study of Small Frozen RAG Modelscited by 1
  34. 2024IntelliCare: Improving Healthcare Analysis with Variance-Controlled Patient-Level Knowledge from Large Language Modelscited by 0
  35. 2024Let your LLM generate a few tokens and you will reduce the need for retrievalcited by 0
  36. 2025Detecting Hallucinations in Authentic LLM-Human Interactionscited by 0
  37. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0
  38. 2026Toward a Gricean Retreat: Probing LLMs for Knowledge Boundaries and Referent Specificitycited by 0

50 most cited of 51 All 51 (JSON)