mlsanityData to 14 Oct 2026

Language Modeling with Editable External Knowledge

arXiv 2406.11830 ↗First posted 17 Jun 2024cs.CL cs.AIcited by 9

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

When the world changes, so does the text that humans write about it. How do we build language models that can be easily updated to reflect these changes? One popular approach is retrieval-augmented generation, in which new documents are inserted into a knowledge base and retrieved during prediction for downstream tasks. Most prior work on these systems have focused on improving behavior during prediction through better retrieval or reasoning. This paper introduces ERASE, which instead improves model behavior when new documents are acquired, by incrementally deleting or rewriting other entries in the knowledge base each time a document is added. In two new benchmark datasets evaluating models' ability to answer questions about a stream of news articles or conversations, ERASE improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute. Code and data are available at https://github.com/belindal/ERASE

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 46 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

32 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 30

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  4. 2020Reformer: The Efficient Transformercited by 1,668
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  7. 2022Mass-Editing Memory in a Transformercited by 1,001
  8. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  9. 2021Editing Factual Knowledge in Language Modelscited by 553
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  11. 2022Memory-Based Model Editing at Scalecited by 447
  12. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
18 more
  1. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  2. 2022RealTime QA: What's the Answer Right Now?cited by 244
  3. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  4. 2021A Dataset for Answering Time-Sensitive Questionscited by 154
  5. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  6. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  7. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  8. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  9. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 67
  10. 2022Temporal Knowledge Graph Completion: A Surveycited by 61
  11. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 56
  12. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  13. 2022DSI++: Updating Transformer Memory with New Documentscited by 35
  14. 2024Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Taskscited by 24
  15. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22
  16. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  17. 2023KILM: Knowledge Injection into Encoder-Decoder Language Modelscited by 19
  18. 2024PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answeringcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 1.

9 papers.

  1. 2025HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generationcited by 13
  2. 2025Memorization vs. Reasoning: Updating LLMs with New Knowledgecited by 13
  3. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  4. 2024STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBackcited by 4
  5. 2025Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMscited by 4
  6. 2025Memorization and Knowledge Injection in Gated LLMscited by 3
  7. 2026From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidencecited by 1
  8. 2024Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probingcited by 0
  9. 2025EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Active Retrieval Augmented Generation
  2. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
  3. 2024E2E-AFG: An End-to-End Model with Adaptive Filtering for Retrieval-Augmented Generation
  4. 2024KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models
  5. 2025Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models
  6. 2024GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  8. 2024Retrieval is Accurate Generation