mlsanityData to 14 Oct 2026

Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering

arXiv 2405.14383 ↗First posted 23 May 2024cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are widely used for knowledge-seeking yet suffer from hallucinations. The knowledge boundary (KB) of an LLM limits its factual understanding, beyond which it may begin to hallucinate. Investigating the perception of LLMs' KB is crucial for detecting hallucinations and LLMs' reliable generation. Current studies perceive LLMs' KB on questions with a concrete answer (close-ended questions) while paying limited attention to semi-open-ended questions (SoeQ) that correspond to many potential answers. Some researchers achieve it by judging whether the question is answerable or not. However, this paradigm is unsuitable for SoeQ, which are usually partially answerable, containing both answerable and ambiguous (unanswerable) answers. Ambiguous answers are essential for knowledge-seeking, but they may go beyond the KB of LLMs. In this paper, we perceive the LLMs' KB with SoeQ by discovering more ambiguous answers. First, we apply an LLM-based approach to construct SoeQ and obtain answers from a target LLM. Unfortunately, the output probabilities of mainstream black-box LLMs are inaccessible to sample for low-probability ambiguous answers. Therefore, we apply an open-sourced auxiliary model to explore ambiguous answers for the target LLM. We calculate the nearest semantic representation for existing answers to estimate their probabilities, with which we reduce the generation probability of high-probability answers to achieve a more effective generation. Finally, we compare the results from the RAG-based evaluation and LLM self-evaluation to categorize four types of ambiguous answers that are beyond the KB of the target LLM. Following our method, we construct a dataset to perceive the KB for GPT-4. We find that GPT-4 performs poorly on SoeQ and is often unaware of its KB. Besides, our auxiliary model, LLaMA-2-13B, is effective in discovering more ambiguous answers.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

35 of 44 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  11. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
23 more
  1. 2022Prompting GPT-3 To Be Reliablecited by 260
  2. 2023Do Large Language Models Know What They Don't Know?cited by 243
  3. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  4. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  5. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  6. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 79
  7. 2023Alignment for Honestycited by 74
  8. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 72
  9. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  10. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  11. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 61
  12. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  13. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 58
  14. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  15. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  16. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  17. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 50
  18. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  19. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 38
  20. 2021Zero-shot Visual Question Answering using Knowledge Graphcited by 37
  21. 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33
  22. 2024Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanationscited by 30
  23. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  2. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  3. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
  4. 2024Distinguishing Ignorance from Error in LLM Hallucinationscited by 14
  5. 2024Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modelingcited by 9
  6. 2025Agentic Knowledgeable Self-awarenesscited by 8
  7. 2025Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundariescited by 7
  8. 2025Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levelscited by 7
  9. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 6
  10. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6
  11. 2025LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertaintycited by 5
  12. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
8 more
  1. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  2. 202460 Data Points are Sufficient to Fine-Tune LLMs for Question-Answeringcited by 2
  3. 2025Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledgecited by 2
  4. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  5. 2025Span-level Detection of AI-generated Scientific Text via Contrastive Learning and Structural Calibrationcited by 0
  6. 2025Assessing the Software Security Comprehension of Large Language Modelscited by 0
  7. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  8. 2026Align Documents to Questions: Question-Oriented Document Rewriting for Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs
  2. 2026Evaluating Hallucinations in Domain-Adapted Large Language Models
  3. 2026KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
  4. 2024Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation
  5. 2024Mitigating LLM Hallucinations via Conformal Abstention
  6. 2025Inside-Out: Hidden Factual Knowledge in LLMs
  7. 2026Deep Learning-based Method for Expressing Knowledge Boundary of Black-Box LLM
  8. 2023Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?