mlsanityData to 14 Oct 2026

Can AI Assistants Know What They Don't Know?

arXiv 2401.13275 ↗First posted 24 Jan 2024cs.CL cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Recently, AI assistants based on large language models (LLMs) show surprising performance in many tasks, such as dialogue, solving math problems, writing code, and using tools. Although LLMs possess intensive world knowledge, they still make factual errors when facing some knowledge intensive tasks, like open-domain question answering. These untruthful responses from the AI assistant may cause significant risks in practical applications. We believe that an AI assistant's refusal to answer questions it does not know is a crucial method for reducing hallucinations and making the assistant truthful. Therefore, in this paper, we ask the question "Can AI assistants know what they don't know and express them through natural language?" To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. Experimental results show that after alignment with Idk datasets, the assistant can refuse to answer most its unknown questions. For questions they attempt to answer, the accuracy is significantly higher than before the alignment.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

38 of 44 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
26 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  10. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  13. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  14. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  15. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  16. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  17. 2023Do Large Language Models Know What They Don't Know?cited by 243
  18. 2023Fine-tuning Language Models for Factualitycited by 221
  19. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  20. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  21. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  22. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  23. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  24. 2023Evaluating Open-QA Evaluationcited by 35
  25. 2023ALCUNA: Large Language Models Meet New Knowledgecited by 35
  26. 2023Evaluating Hallucinations in Chinese Large Language Modelscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 20; 2026: 14.

50 papers.

Sort:

By area:

  1. 2024Language Models with Conformal Factuality Guaranteescited by 131
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  4. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  5. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  6. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  7. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  8. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 46
  9. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  10. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  11. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  12. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
38 more
  1. 2024A Survey on the Honesty of Large Language Modelscited by 23
  2. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  3. 2024Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Modelscited by 19
  4. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  5. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  6. 2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
  7. 2024Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropycited by 11
  8. 2025Do Large Language Models Know What They Are Capable Of?cited by 10
  9. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  10. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
  11. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  12. 2025AI Awarenesscited by 7
  13. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 7
  14. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 6
  15. 2025Can LLMs Refuse Questions They Do Not Know? Measuring Knowledge-Aware Refusal in Factual Taskscited by 6
  16. 2025Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinationscited by 5
  17. 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answeringcited by 5
  18. 2025Measuring Epistemic Humility in Multimodal Large Language Modelscited by 4
  19. 2026FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximizationcited by 4
  20. 2024Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editingcited by 3
  21. 2025When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMscited by 3
  22. 2026When Should LLMs Search? Counterfactual Supervision for Search Routingcited by 3
  23. 2025High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuningcited by 2
  24. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systemscited by 2
  25. 2026Do Large Language Models Know What They Don't Know II? A Fully Behavioral, Non-Cognitive Measure of Epistemic Honestycited by 2
  26. 2024Reducing the Scope of Language Modelscited by 1
  27. 2024Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusalcited by 1
  28. 2025Teaming LLMs to Detect and Mitigate Hallucinationscited by 1
  29. 2026UCPO: Uncertainty-Aware Policy Optimizationcited by 1
  30. 2026Delineating Knowledge Boundaries for Honest Large Vision-Language Modelscited by 1
  31. 2024Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantizationcited by 0
  32. 2026Only Say What You Know: Calibration-Aware Generation for Long-Form Factualitycited by 0
  33. 2026ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learningcited by 0
  34. 2026Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoningcited by 0
  35. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  36. 2026ChatT2: An Adaptive Framework for Developing a Large Language Model-Based Agent for Natural Product Domain Researchcited by 0
  37. 2026Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspectivecited by 0
  38. 2026Strong yet Controllable LLM Abstention without Learning to Abstaincited by 0