# Language Models are Open Knowledge Graphs

arXiv: https://arxiv.org/abs/2010.11967 · HTML: https://mlsanity.com/paper/2010.11967/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 30 of 95)

- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs](https://mlsanity.com/paper/2305.12788/) (2023)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization](https://mlsanity.com/paper/2105.12544/) (2021)
- [DeepStruct: Pretraining of Language Models for Structure Prediction](https://mlsanity.com/paper/2205.10475/) (2022)
- [Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning](https://mlsanity.com/paper/2312.17484/) (2023)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphs](https://mlsanity.com/paper/2410.02811/) (2024)
- [PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning](https://mlsanity.com/paper/2305.13888/) (2023)
- [Iterative Zero-Shot LLM Prompting for Knowledge Graph Construction](https://mlsanity.com/paper/2307.01128/) (2023)
- [Robust Retrieval Augmented Generation for Zero-shot Slot Filling](https://mlsanity.com/paper/2108.13934/) (2021)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [FolkScope: Intention Knowledge Graph Construction for E-commerce Commonsense Discovery](https://mlsanity.com/paper/2211.08316/) (2022)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Zero-Shot Information Extraction as a Unified Text-to-Triple Translation](https://mlsanity.com/paper/2109.11171/) (2021)
- [Measuring Social Norms of Large Language Models](https://mlsanity.com/paper/2404.02491/) (2024)
- [OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Models](https://mlsanity.com/paper/2407.11213/) (2024)
- [SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models](https://mlsanity.com/paper/2411.02433/) (2024)

## Abstract (arXiv)

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.