# Language Modeling with Editable External Knowledge

arXiv: https://arxiv.org/abs/2406.11830 · HTML: https://mlsanity.com/paper/2406.11830/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 30 of 32)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge](https://mlsanity.com/paper/2305.01651/) (2023)
- [Temporal Knowledge Graph Completion: A Survey](https://mlsanity.com/paper/2201.08236/) (2022)
- [KG-RAG: Bridging the Gap Between Knowledge and Creativity](https://mlsanity.com/paper/2405.12035/) (2024)
- [Entity Cloze By Date: What LMs Know About Unseen Entities](https://mlsanity.com/paper/2205.02832/) (2022)
- [DSI++: Updating Transformer Memory with New Documents](https://mlsanity.com/paper/2212.09744/) (2022)
- [Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks](https://mlsanity.com/paper/2401.17585/) (2024)
- [Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability](https://mlsanity.com/paper/2401.08574/) (2024)
- [Mitigating Temporal Misalignment by Discarding Outdated Facts](https://mlsanity.com/paper/2305.14824/) (2023)

## Built on by (most cited first, 9 of 9)

- [HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.04800/) (2025)
- [Memorization vs. Reasoning: Updating LLMs with New Knowledge](https://mlsanity.com/paper/2504.12523/) (2025)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBack](https://mlsanity.com/paper/2410.10584/) (2024)
- [Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs](https://mlsanity.com/paper/2506.07270/) (2025)
- [Memorization and Knowledge Injection in Gated LLMs](https://mlsanity.com/paper/2504.21239/) (2025)
- [From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence](https://mlsanity.com/paper/2605.06006/) (2026)
- [Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing](https://mlsanity.com/paper/2408.14849/) (2024)
- [EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge](https://mlsanity.com/paper/2507.03617/) (2025)

## Abstract (arXiv)

When the world changes, so does the text that humans write about it. How do we build language models that can be easily updated to reflect these changes? One popular approach is retrieval-augmented generation, in which new documents are inserted into a knowledge base and retrieved during prediction for downstream tasks. Most prior work on these systems have focused on improving behavior during prediction through better retrieval or reasoning. This paper introduces ERASE, which instead improves model behavior when new documents are acquired, by incrementally deleting or rewriting other entries in the knowledge base each time a document is added. In two new benchmark datasets evaluating models' ability to answer questions about a stream of news articles or conversations, ERASE improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute. Code and data are available at https://github.com/belindal/ERASE