# Are LLMs Really Not Knowledgeable? Mining the Submerged Knowledge in LLMs' Memory

arXiv: https://arxiv.org/abs/2412.20846 · HTML: https://mlsanity.com/paper/2412.20846/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)
- [Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments](https://mlsanity.com/paper/2212.09736/) (2022)
- [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023)
- [Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation](https://mlsanity.com/paper/2402.11493/) (2024)
- [HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild](https://mlsanity.com/paper/2403.04307/) (2024)
- [Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?](https://mlsanity.com/paper/2311.09702/) (2023)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Quantifying and Attributing the Hallucination of Large Language Models via Association Analysis](https://mlsanity.com/paper/2309.05217/) (2023)
- [Automatic Question-Answer Generation for Long-Tail Knowledge](https://mlsanity.com/paper/2403.01382/) (2024)

## Built on by (most cited first, 1 of 1)

- [Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignment](https://mlsanity.com/paper/2512.09148/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have shown promise as parametric knowledge bases, but often underperform on question answering (QA) tasks due to hallucinations and uncertainty. While prior work attributes these failures to knowledge gaps in the model's parameters, we uncover a complementary phenomenon: LLMs frequently retain correct knowledge even when generating incorrect or "unsure" answers. By analyzing the token-level output distributions, we find that correct answers often appear among high-probability candidates, despite not being selected. Motivated by this, we propose Hits@k, a novel metric to evaluate latent knowledge retention independent of answer surface form. Our experiments reveal that LLMs possess significantly more factual knowledge than is reflected by standard QA accuracy. Building on these insights, we further examine the prevailing few-shot QA paradigm. We find that prompting strategies which allow "unsure" outputs can inadvertently suppress correct answers by discouraging low-confidence generation. We design a set of quantitative experiments to measure this suppression effect, offering practical guidance for future prompt and decoding design in knowledge-intensive tasks.