mlsanityData to 7 Oct 2026

Relational Memory Augmented Language Models

arXiv 2201.09680 ↗First posted 24 Jan 2022cs.CL cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model with a knowledge graph for a more coherent and logical generation.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 62 of its 99 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

62 of 99 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2019Language Models as Knowledge Bases?cited by 1,922
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2016Complex Embeddings for Simple Link Predictioncited by 897
  6. 2018Link Prediction Based on Graph Neural Networkscited by 829
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  9. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  10. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  11. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  12. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  13. 2020Autoregressive Entity Retrievalcited by 333
  14. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
  15. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  16. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  17. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  18. 2020Nearest Neighbor Machine Translationcited by 209
  19. 2019Episodic Memory in Lifelong Language Learningcited by 188
  20. 2019Quaternion Knowledge Graph Embeddingscited by 175
  21. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  22. 2018DKN: Deep Knowledge-Aware Network for News Recommendationcited by 137
  23. 2020Language Models are Open Knowledge Graphscited by 95
  24. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  25. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  26. 2021Adaptive Semiparametric Language Modelscited by 83
  27. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  28. 2019Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modelingcited by 71
  29. 2019Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendationcited by 70
  30. 2016Efficient softmax approximation for GPUscited by 67
  31. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  32. 2020Augmenting Transformers with KNN-Based Composite Memory for Dialoguecited by 53
  33. 2020Word meaning in minds and machinescited by 51
  34. 2019Leveraging Knowledge Bases in LSTMs for Improving Machine Readingcited by 37
  35. 2019Non-Parametric Adaptation for Neural Machine Translationcited by 34
  36. 2021Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialogcited by 33
  37. 2019Differentiable Reasoning on Large Knowledge Bases and Natural Languagecited by 30
  38. 2020Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-trainingcited by 24

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 10; 2024: 6; 2025: 2.

24 papers.

Sort:

By area:

  1. 2022A Review on Language Models as Knowledge Basescited by 156
  2. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  3. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  4. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  5. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  6. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  7. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  8. 2022Retrieval-based Controllable Molecule Generationcited by 44
  9. 2023YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomycited by 37
  10. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  11. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  12. 2025Cognitive Memory in Large Language Modelscited by 28
12 more
  1. 2023Dissociating language and thought in large language modelscited by 27
  2. 2023A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promisescited by 13
  3. 2024Similarity-based Neighbor Selection for Graph LLMscited by 10
  4. 2024Writing in the Margins: Better Inference Pattern for Long Context Retrievalcited by 7
  5. 2023Similarity-weighted Construction of Contextualized Commonsense Knowledge Graphs for Knowledge-intense Argumentation Taskscited by 6
  6. 2025AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbotcited by 5
  7. 2022Analogical Math Word Problems Solving with Enhanced Problem-Solution Associationcited by 4
  8. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  9. 2024Online Training of Large Language Models: Learn while chattingcited by 3
  10. 2022"Covid vaccine is against Covid but Oxford vaccine is made at Oxford!" Semantic Interpretation of Proper Noun Compoundscited by 1
  11. 2023Advancing Perception in Artificial Intelligence through Principles of Cognitive Sciencecited by 1
  12. 2024Memory-Augmented Generative Adversarial Transformerscited by 1