mlsanityData to 14 Oct 2026

Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing

arXiv 2505.21701 ↗First posted 27 May 2025cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

The reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs. Various methods for probing such gaps exist, ranging from calibration-based to prompting-based methods. To evaluate these probing methods, in this paper, we propose a new process based on using input variations and quantitative metrics. Through this, we expose two dimensions of inconsistency in knowledge gap probing. (1) Intra-method inconsistency: Minimal non-semantic perturbations in prompts lead to considerable variance in detected knowledge gaps within the same probing method; e.g., the simple variation of shuffling answer options can decrease agreement to around 40%. (2) Cross-method inconsistency: Probing methods contradict each other on whether a model knows the answer. Methods are highly inconsistent -- with decision consistency across methods being as low as 7% -- even though the model, dataset, and prompt are all the same. These findings challenge existing probing methods and highlight the urgent need for perturbation-robust probing frameworks.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

31 of 50 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
19 more
  1. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  2. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  3. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  4. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  5. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  6. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  7. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  8. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 92
  9. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  10. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  11. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  12. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  13. 2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 55
  14. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  15. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  16. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  17. 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33
  18. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  19. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languagescited by 8
  2. 2026A Unified Framework to Quantify Cultural Intelligence of AIcited by 2
  3. 2025Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samplescited by 1
  4. 2026Autolearn: Learn by Surprise, Commit by Proofcited by 0
  5. 2026ReverseMath: Answer Inversion for Scalable and Verifiable Mathematical Problem Generationcited by 0
  6. 2026Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesiscited by 0
  7. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  8. 2026Do correct answers have stable latent knowledge support?cited by 0