# Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling

arXiv: https://arxiv.org/abs/1906.07241 · HTML: https://mlsanity.com/paper/1906.07241/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2019.

## Built on by (most cited first, 30 of 71)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering](https://mlsanity.com/paper/2309.11206/) (2023)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Rethinking Explainability as a Dialogue: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01875/) (2022)
- [RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding](https://mlsanity.com/paper/2212.01588/) (2022)
- [Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey](https://mlsanity.com/paper/2201.12438/) (2022)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations](https://mlsanity.com/paper/2403.03008/) (2024)
- [Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models](https://mlsanity.com/paper/2106.01623/) (2021)
- [Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation](https://mlsanity.com/paper/2110.05456/) (2021)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [Graph-Based Retriever Captures the Long Tail of Biomedical Knowledge](https://mlsanity.com/paper/2402.12352/) (2024)
- [Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation](https://mlsanity.com/paper/2101.00760/) (2021)
- [KILM: Knowledge Injection into Encoder-Decoder Language Models](https://mlsanity.com/paper/2302.09170/) (2023)
- [Reasoning Over Virtual Knowledge Bases With Open Predicate Relations](https://mlsanity.com/paper/2102.07043/) (2021)
- [Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?](https://mlsanity.com/paper/2109.11321/) (2021)
- [Visually-Augmented Language Modeling](https://mlsanity.com/paper/2205.10178/) (2022)
- [Exploiting Sentiment and Common Sense for Zero-shot Stance Detection](https://mlsanity.com/paper/2208.08797/) (2022)
- [Knowledge Infused Decoding](https://mlsanity.com/paper/2204.03084/) (2022)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Graph-Text Multi-Modal Pre-training for Medical Representation Learning](https://mlsanity.com/paper/2203.09994/) (2022)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)

## Abstract (arXiv)

Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge. However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them. To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. We also introduce the Linked WikiText-2 dataset, a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark. In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model. We additionally compare different language model's ability to complete sentences requiring factual knowledge, showing that the KGLM outperforms even very large language models in generating facts.