mlsanityData to 7 Oct 2026

Defining Knowledge: Bridging Epistemology and Large Language Models

arXiv 2410.02499 ↗First posted 3 Oct 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Knowledge claims are abundant in the literature on large language models (LLMs); but can we say that GPT-4 truly "knows" the Earth is round? To address this question, we review standard definitions of knowledge in epistemology and we formalize interpretations applicable to LLMs. In doing so, we identify inconsistencies and gaps in how current NLP research conceptualizes knowledge with respect to epistemological frameworks. Additionally, we conduct a survey of 100 professional philosophers and computer scientists to compare their preferences in knowledge definitions and their views on whether LLMs can really be said to know. Finally, we suggest evaluation protocols for testing knowledge in accordance to the most relevant definitions.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 71 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

37 of 71 references.

References · 37

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2022Mass-Editing Memory in a Transformercited by 976
  5. 2019How Can We Know What Language Models Know?cited by 953
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  7. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  8. 2021Editing Factual Knowledge in Language Modelscited by 548
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  12. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
25 more
  1. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  2. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  3. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  4. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  5. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  6. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  7. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  8. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  9. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  10. 2024To Believe or Not to Believe Your LLMcited by 120
  11. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  12. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  13. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63
  14. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 56
  15. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  16. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  17. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  18. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  19. 2023Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationalescited by 37
  20. 2024Locating and Editing Factual Associations in Mambacited by 35
  21. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  22. 2023Challenges with unsupervised LLM knowledge discoverycited by 30
  23. 2024Learning to Plan and Generate Text with Citationscited by 28
  24. 2024MuLan: A Study of Fact Mutability in Language Modelscited by 10
  25. 2024Making AI Intelligible: Philosophical Foundationscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 6.

7 papers.

  1. 2025Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questionscited by 9
  2. 2025Mechanistic Interpretability Needs Philosophycited by 6
  3. 2024Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completioncited by 4
  4. 2025RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Modelscited by 4
  5. 2025Fine-Grained Detection of Context-Grounded Hallucinations Using LLMscited by 1
  6. 2025Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Researchcited by 0
  7. 2025Epistemoverse: Toward an AI-Driven Knowledge Metaverse for Intellectual Heritage Preservationcited by 0