mlsanityData to 14 Oct 2026

Pre-training Limited Memory Language Models with Internal and External Knowledge

arXiv 2505.15962 ↗First posted 21 May 2025cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Neural language models are black-boxes--both linguistic patterns and factual knowledge are distributed across billions of opaque parameters. This entangled encoding makes it difficult to reliably inspect, verify, or update specific facts. We introduce Limited Memory Language Models (LMLM), a new class of language models that externalizes factual knowledge to external database during pre-training rather than memorizing them. Our pre-training approach strategically masks externally retrieved factual values from the training loss, thereby teaching the model to perform targeted lookups rather than relying on memorization in model weights. Our experiments demonstrate that LMLMs achieve competitive performance compared to significantly larger LLMs on standard benchmarks, while offering the advantages of explicit, editable, and verifiable knowledge bases.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 56 of its 69 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

56 of 69 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2019Language Models as Knowledge Bases?cited by 1,936
38 more
  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  9. 2022Mass-Editing Memory in a Transformercited by 1,001
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  12. 2023Large Language Modelscited by 780
  13. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  14. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  15. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  16. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  17. 2021Fast Model Editing at Scalecited by 498
  18. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  19. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  20. 2024Measuring short-form factuality in large language modelscited by 373
  21. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  22. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  23. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  24. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  25. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  26. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  27. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  28. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  29. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  30. 2020Editable Neural Networkscited by 155
  31. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  32. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  33. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 101
  34. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  35. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  36. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  37. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 73
  38. 2024Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?cited by 48

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  3. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  4. 2026Cram Less to Fit More: Training Data Pruning Improves Memorization of Factscited by 3
  5. 2026Auditing Forgetting in Limited Memory Language Modelscited by 2
  6. 2026Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memorycited by 1
  7. 2025Rotation Control Unlearning: Quantifying and Controlling Continuous Unlearning for LLM with The Cognitive Rotation Spacecited by 0
  8. 2026LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Losscited by 0
  9. 2026Autolearn: Learn by Surprise, Commit by Proofcited by 0
  10. 2026MemDefrag: Latent Memory Defragmentation for Large Language Modelscited by 0
  11. 2026Co-LMLM: Continuous-Query Limited Memory Language Modelscited by 0
  12. 2026Co-Evolving Structured Knowledge and Reasoning in Language Modelscited by 0