mlsanityData to 7 Oct 2026

Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases

arXiv 2106.09231 ↗First posted 17 Jun 2021cs.CL cs.AIcited by 107

Not on the map. Why?

Abstract (arXiv)

Previous literatures show that pre-trained masked language models (MLMs) such as BERT can achieve competitive factual knowledge extraction performance on some datasets, indicating that MLMs can potentially be a reliable knowledge source. In this paper, we conduct a rigorous study to explore the underlying predicting mechanisms of MLMs over different extraction paradigms. By investigating the behaviors of MLMs, we find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts. Furthermore, incorporating illustrative cases and external contexts improve knowledge prediction mainly due to entity type guidance and golden answer leakage. Our findings shed light on the underlying predicting mechanisms of MLMs, and strongly question the previous conclusion that current MLMs can potentially serve as reliable factual knowledge bases.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

38 of 51 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  12. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
26 more
  1. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  2. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  3. 2016Neural Architectures for Named Entity Recognitioncited by 548
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  6. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  7. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  8. 2019Assessing BERT's Syntactic Abilitiescited by 191
  9. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  10. 2020How Context Affects Language Models' Factual Predictionscited by 181
  11. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  12. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  13. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  14. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  15. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  16. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  17. 2020Language Models are Open Knowledge Graphscited by 95
  18. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  19. 2019Inducing Relational Knowledge from BERTcited by 75
  20. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  21. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  22. 2020Language Models as Few-Shot Learner for Task-Oriented Dialogue Systemscited by 41
  23. 2020"You are grounded!": Latent Name Artifacts in Pre-trained Language Modelscited by 39
  24. 2020BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QAcited by 36
  25. 2020Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoningcited by 33
  26. 2021Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 22; 2023: 29; 2024: 27; 2025: 13; 2026: 5.

107 papers.

Sort:

By area:

  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  3. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  4. 2023PMET: Precise Model Editing in a Transformercited by 170
  5. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  6. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  7. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  10. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  11. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  12. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
38 more
  1. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  2. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  4. 2023Do Large Language Models Know about Facts?cited by 49
  5. 2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45
  6. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  7. 2022How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysiscited by 42
  8. 2023Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questionscited by 42
  9. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
  10. 2024Knowledge Graph Enhanced Large Language Model Editingcited by 38
  11. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 35
  12. 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
  13. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  14. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  15. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  16. 2023Editing Language Model-based Knowledge Graph Embeddingscited by 23
  17. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  18. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
  19. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  20. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  21. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  22. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19
  23. 2025FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Modelscited by 19
  24. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  25. 2023Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Modelscited by 18
  26. 2023DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Modelscited by 17
  27. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  28. 2022Neural Knowledge Bank for Pretrained Transformerscited by 16
  29. 2023Measuring and Modifying Factual Knowledge in Large Language Modelscited by 16
  30. 2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
  31. 2021DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understandingcited by 15
  32. 2022Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probingcited by 15
  33. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  34. 2023Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awarenesscited by 14
  35. 2023Assessing the Reliability of Large Language Model Knowledgecited by 13
  36. 2024Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extractioncited by 13
  37. 2024BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learningcited by 13
  38. 2023Understanding Finetuning for Factual Knowledge Extraction from Language Modelscited by 12

50 most cited of 107 All 107 (JSON)