# How Relevant is Selective Memory Population in Lifelong Language Learning?

arXiv: https://arxiv.org/abs/2210.00940 · HTML: https://mlsanity.com/paper/2210.00940/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [Efficient Meta Lifelong-Learning with Limited Memory](https://mlsanity.com/paper/2010.02500/) (2020)
- [Information-theoretic Online Memory Selection for Continual Learning](https://mlsanity.com/paper/2204.04763/) (2022)
- [Memory-based Parameter Adaptation](https://mlsanity.com/paper/1802.10542/) (2018)
- [Meta-Learning with Sparse Experience Replay for Lifelong Language Learning](https://mlsanity.com/paper/2009.04891/) (2020)
- [Selective Replay Enhances Learning in Online Continual Analogical Reasoning](https://mlsanity.com/paper/2103.03987/) (2021)
- [Entropy-based Stability-Plasticity for Lifelong Learning](https://mlsanity.com/paper/2204.09517/) (2022)

## Built on by (most cited first, 6 of 6)

- [Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models](https://mlsanity.com/paper/2408.09053/) (2024)
- [Continual Learning for Predictive Maintenance: Overview and Challenges](https://mlsanity.com/paper/2301.12467/) (2023)
- [Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocation](https://mlsanity.com/paper/2310.00154/) (2023)
- [SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning](https://mlsanity.com/paper/2511.22367/) (2025)
- [In-context Interference in Chat-based Large Language Models](https://mlsanity.com/paper/2309.12727/) (2023)
- [Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights](https://mlsanity.com/paper/2604.17578/) (2026)

## Abstract (arXiv)

Lifelong language learning seeks to have models continuously learn multiple tasks in a sequential order without suffering from catastrophic forgetting. State-of-the-art approaches rely on sparse experience replay as the primary approach to prevent forgetting. Experience replay usually adopts sampling methods for the memory population; however, the effect of the chosen sampling strategy on model performance has not yet been studied. In this paper, we investigate how relevant the selective memory population is in the lifelong learning process of text classification and question-answering tasks. We found that methods that randomly store a uniform number of samples from the entire data stream lead to high performances, especially for low memory size, which is consistent with computer vision studies.