# A Neural Knowledge Language Model

arXiv: https://arxiv.org/abs/1608.00318 · HTML: https://mlsanity.com/paper/1608.00318/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2016.

## Built on by (most cited first, 13 of 13)

- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Relational Memory Augmented Language Models](https://mlsanity.com/paper/2201.09680/) (2022)
- [Reasoning Over Virtual Knowledge Bases With Open Predicate Relations](https://mlsanity.com/paper/2102.07043/) (2021)
- [Pretraining with hierarchical memories: separating long-tail and common knowledge](https://mlsanity.com/paper/2510.02375/) (2025)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web](https://mlsanity.com/paper/2102.09681/) (2021)
- [Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU](https://mlsanity.com/paper/2106.09589/) (2021)
- [TopNet: Learning from Neural Topic Model to Generate Long Stories](https://mlsanity.com/paper/2112.07259/) (2021)
- [Contextual Text Embeddings for Twi](https://mlsanity.com/paper/2103.15963/) (2021)
- [SubGraph Networks based Entity Alignment for Cross-lingual Knowledge Graph](https://mlsanity.com/paper/2205.03557/) (2022)

## Abstract (arXiv)

Current language models have a significant limitation in the ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge from statistical co-occurrences although most of the knowledge words are rarely observed. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by the knowledge graph with the RNN language model. By predicting whether the word to generate has an underlying fact or not, the model can generate such knowledge-related words by copying from the description of the predicted fact. In experiments, we show that the NKLM significantly improves the performance while generating a much smaller number of unknown words.