# Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models

arXiv: https://arxiv.org/abs/2305.09144 · HTML: https://mlsanity.com/paper/2305.09144/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 75 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [A comprehensive, application-oriented study of catastrophic forgetting in DNNs](https://mlsanity.com/paper/1905.08101/) (2019)
- [Memorisation versus Generalisation in Pre-trained Language Models](https://mlsanity.com/paper/2105.00828/) (2021)
- [Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View](https://mlsanity.com/paper/2203.12258/) (2022)

## Built on by (most cited first, 18 of 18)

- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [How Much Can We Forget about Data Contamination?](https://mlsanity.com/paper/2410.03249/) (2024)
- [StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation](https://mlsanity.com/paper/2408.03281/) (2024)
- [Revealing and Mitigating Over-Attention in Knowledge Editing](https://mlsanity.com/paper/2502.14838/) (2025)
- [Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models](https://mlsanity.com/paper/2511.06793/) (2025)
- [Order-Robust Class Incremental Learning: Graph-Driven Dynamic Similarity Grouping](https://mlsanity.com/paper/2502.20032/) (2025)
- [Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs](https://mlsanity.com/paper/2512.14741/) (2025)
- [Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining](https://mlsanity.com/paper/2509.05291/) (2025)
- [Train Once, Answer All: Many Pretraining Experiments for the Cost of One](https://mlsanity.com/paper/2509.23383/) (2025)
- [Do Your Best and Get Enough Rest for Continual Learning](https://mlsanity.com/paper/2503.18371/) (2025)
- [Derivational Morphology Reveals Analogical Generalization in Large Language Models](https://mlsanity.com/paper/2411.07990/) (2024)
- [An Efficient Replay for Class-Incremental Learning with Pre-trained Models](https://mlsanity.com/paper/2408.08084/) (2024)
- [The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?](https://mlsanity.com/paper/2509.22580/) (2025)
- [Open Problems and a Hypothetical Path Forward in LLM Knowledge Paradigms](https://mlsanity.com/paper/2504.06823/) (2025)
- [Realistic Continual Learning Approach using Pre-trained Models](https://mlsanity.com/paper/2404.07729/) (2024)

## Abstract (arXiv)

Memory is one of the most essential cognitive functions serving as a repository of world knowledge and episodes of activities. In recent years, large-scale pre-trained language models have shown remarkable memorizing ability. On the contrary, vanilla neural networks without pre-training have been long observed suffering from the catastrophic forgetting problem. To investigate such a retentive-forgetful contradiction and understand the memory mechanism of language models, we conduct thorough experiments by controlling the target knowledge types, the learning strategies and the learning schedules. We find that: 1) Vanilla language models are forgetful; 2) Pre-training leads to retentive language models; 3) Knowledge relevance and diversification significantly influence the memory formation. These conclusions are useful for understanding the abilities of pre-trained language models and shed light on designing and evaluating new learning and inference algorithms of language models.