mlsanityData to 14 Oct 2026

Inducing Relational Knowledge from BERT

arXiv 1911.12753 ↗First posted 28 Nov 2019cs.CL cs.AIcited by 75

Not on the map. Why?

Abstract (arXiv)

One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 21; 2023: 12; 2024: 9; 2025: 5; 2026: 2.

75 papers.

Sort:

By area:

  1. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  4. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  5. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  6. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  7. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  8. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  9. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  10. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  11. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 89
  12. 2021BERTese: Learning to Speak to BERTcited by 82
38 more
  1. 2021A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutionscited by 78
  2. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  3. 2021TIMEDIAL: Temporal Commonsense Reasoning in Dialogcited by 55
  4. 2021Complex Knowledge Base Question Answering: A Surveycited by 52
  5. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  6. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  7. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  8. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  9. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  10. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  11. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  12. 2022BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Modelscited by 20
  13. 2021Refined Commonsense Knowledge from Large-Scale Web Contentscited by 19
  14. 2021What do Large Language Models Learn about Scripts?cited by 19
  15. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  16. 2021Distilling Relation Embeddings from Pre-trained Language Modelscited by 15
  17. 2022Factual Consistency of Multilingual Pretrained Language Modelscited by 15
  18. 2021ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicalitycited by 12
  19. 2021Static Embeddings as Efficient Knowledge Bases?cited by 11
  20. 2021Probing Pre-Trained Language Models for Disease Knowledgecited by 10
  21. 2023Traffic-Domain Video Question Answering with Automatic Captioningcited by 10
  22. 2023On the Evolution of Knowledge Graphs: A Survey and Perspectivecited by 10
  23. 2021Language Models for Lexical Inference in Contextcited by 9
  24. 2021PaCo: Preconditions Attributed to Commonsense Knowledgecited by 9
  25. 2022Modelling Commonsense Properties using Pre-Trained Bi-Encoderscited by 9
  26. 2021Semi-Supervised Exaggeration Detection of Health Science Press Releasescited by 8
  27. 2022Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Inductioncited by 8
  28. 2022Can Language Models Be Specific? How?cited by 8
  29. 2024Beyond Memorization: The Challenge of Random Memory Access in Language Modelscited by 8
  30. 2024Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extractioncited by 7
  31. 2021Probing Linguistic Information For Logical Inference In Pre-trained Language Modelscited by 6
  32. 2021Classifying Dyads for Militarized Conflict Analysiscited by 5
  33. 2022What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structuredcited by 5
  34. 2022SPE: Symmetrical Prompt Enhancement for Fact Probingcited by 5
  35. 2024Comparing Template-based and Template-free Language Model Probingcited by 5
  36. 2025An Investigation of Memorization Risk in Healthcare Foundation Modelscited by 5
  37. 2022Probing Script Knowledge from Pre-Trained Modelscited by 4
  38. 2023LLM2KB: Constructing Knowledge Bases using instruction tuned context aware Large Language Modelscited by 4

50 most cited of 75 All 75 (JSON)