Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
Not on the map. Why?
Abstract (arXiv)
Understanding how large language models (LLMs) acquire, retain, and apply knowledge remains an open challenge. This paper introduces a novel framework, K-(CSA)^2, which categorizes LLM knowledge along two dimensions: correctness and confidence. The framework defines six categories of knowledge, ranging from highly confident correctness to confidently held misconceptions, enabling a nuanced evaluation of model comprehension beyond binary accuracy. Using this framework, we demonstrate how techniques like chain-of-thought prompting and reinforcement learning with human feedback fundamentally alter the knowledge structures of internal (pre-trained) and external (context-dependent) knowledge in LLMs. CoT particularly enhances base model performance and shows synergistic benefits when applied to aligned LLMs. Moreover, our layer-wise analysis reveals that higher layers in LLMs encode more high-confidence knowledge, while low-confidence knowledge tends to emerge in middle-to-lower layers.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 2 Jan 2025.
Builds on
20 of 29 references.
References · 20
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2024Qwen2 Technical Reportcited by 2,076
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2023The Internal State of an LLM Knows When It's Lyingcited by 644
- 2024Measuring short-form factuality in large language modelscited by 345
- 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
- 2023Do Large Language Models Know What They Don't Know?cited by 238
- 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
8 more
- 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
- 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
- 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
- 2024Large language models surpass human experts in predicting neuroscience resultscited by 69
- 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
- 2024KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checkingcited by 26
- 2024Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methodscited by 9
- 2023KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Towards Uncovering How Large Language Model Works: An Explainability Perspective
- 2026Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thought
- 2024Disentangling Memory and Reasoning Ability in Large Language Models
- 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
- 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
- 2024LLM2: Let Large Language Models Harness System 2 Reasoning
- 2026Enhancing reasoning accuracy in large language models during inference time
- 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback