mlsanityData to 7 Oct 2026

Language Models are Open Knowledge Graphs

arXiv 2010.11967 ↗First posted 22 Oct 2020cs.CL cs.AI cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 20; 2023: 27; 2024: 18; 2025: 9; 2026: 3.

95 papers.

Sort:

By area:

  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2021GPT Understands, Toocited by 715
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  5. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  6. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2022A Review on Language Models as Knowledge Basescited by 156
  9. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  10. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  11. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  12. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
38 more
  1. 2021Towards Zero-Label Language Learningcited by 76
  2. 2023GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphscited by 57
  3. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  4. 2021Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarizationcited by 46
  5. 2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45
  6. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  7. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  8. 2024SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphscited by 31
  9. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  10. 2023Iterative Zero-Shot LLM Prompting for Knowledge Graph Constructioncited by 25
  11. 2021Robust Retrieval Augmented Generation for Zero-shot Slot Fillingcited by 24
  12. 2022Relational Memory Augmented Language Modelscited by 24
  13. 2022FolkScope: Intention Knowledge Graph Construction for E-commerce Commonsense Discoverycited by 23
  14. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  15. 2021Zero-Shot Information Extraction as a Unified Text-to-Triple Translationcited by 22
  16. 2024Measuring Social Norms of Large Language Modelscited by 16
  17. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  18. 2021DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understandingcited by 15
  19. 2022Knowledge Graph Generation From Textcited by 15
  20. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 15
  21. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  22. 2024Measuring Vision-Language STEM Skills of Neural Modelscited by 14
  23. 2025Language Models' Factuality Depends on the Language of Inquirycited by 14
  24. 2021Prix-LM: Pretraining for Multilingual Knowledge Base Constructioncited by 12
  25. 2023"Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphscited by 11
  26. 2021Predicting Discourse Trees from Transformer-based Neural Summarizerscited by 10
  27. 2022Entity-aware Transformers for Entity Searchcited by 10
  28. 2023Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Modelscited by 10
  29. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
  30. 2023Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMscited by 9
  31. 2022A Review of Knowledge Graph Completioncited by 8
  32. 2022Can Language Models Be Specific? How?cited by 8
  33. 2022Benchmarking Language Models for Code Syntax Understandingcited by 8
  34. 2023Exploring and Verbalizing Academic Ideas by Concept Co-occurrencecited by 8
  35. 2021Open Relation Modeling: Learning to Define Relations between Entitiescited by 7
  36. 2023A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classificationcited by 7
  37. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  38. 2023EpiK-Eval: Evaluation for Language Models as Epistemic Modelscited by 6

50 most cited of 95 All 95 (JSON)