# Information-theoretic Online Memory Selection for Continual Learning

arXiv: https://arxiv.org/abs/2204.04763 · HTML: https://mlsanity.com/paper/2204.04763/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Functional Regularisation for Continual Learning with Gaussian Processes](https://mlsanity.com/paper/1901.11356/) (2019)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Continual Deep Learning by Functional Regularisation of Memorable Past](https://mlsanity.com/paper/2004.14070/) (2020)
- [Imbalanced Continual Learning with Partitioning Reservoir Sampling](https://mlsanity.com/paper/2009.03632/) (2020)
- [Automated Scalable Bayesian Inference via Hilbert Coresets](https://mlsanity.com/paper/1710.05053/) (2017)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Streaming Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1705.07131/) (2017)
- [The Effectiveness of Memory Replay in Large Scale Continual Learning](https://mlsanity.com/paper/2010.02418/) (2020)
- [Online Submodular Maximization with Preemption](https://mlsanity.com/paper/1501.05801/) (2015)

## Built on by (most cited first, 30 of 42)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Computationally Budgeted Continual Learning: What Does Matter?](https://mlsanity.com/paper/2303.11165/) (2023)
- [Real-Time Evaluation in Online Continual Learning: A New Hope](https://mlsanity.com/paper/2302.01047/) (2023)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Online Continual Learning Without the Storage Constraint](https://mlsanity.com/paper/2305.09253/) (2023)
- [Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities](https://mlsanity.com/paper/2208.00549/) (2022)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Task-Free Continual Learning via Online Discrepancy Distance Learning](https://mlsanity.com/paper/2210.06579/) (2022)
- [Streaming Active Learning with Deep Neural Networks](https://mlsanity.com/paper/2303.02535/) (2023)
- [Summarizing Stream Data for Memory-Constrained Online Continual Learning](https://mlsanity.com/paper/2305.16645/) (2023)
- [Kalman Filter for Online Classification of Non-Stationary Data](https://mlsanity.com/paper/2306.08448/) (2023)
- [Optimizing Class Distribution in Memory for Multi-Label Online Continual Learning](https://mlsanity.com/paper/2209.11469/) (2022)
- [Learn the Time to Learn: Replay Scheduling in Continual Learning](https://mlsanity.com/paper/2209.08660/) (2022)
- [Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective](https://mlsanity.com/paper/2506.00205/) (2025)
- [Memory Population in Continual Learning via Outlier Elimination](https://mlsanity.com/paper/2207.01145/) (2022)
- [Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompt](https://mlsanity.com/paper/2310.13024/) (2023)
- [Knowledge Restore and Transfer for Multi-label Class-Incremental Learning](https://mlsanity.com/paper/2302.13334/) (2023)
- [Improving Replay Sample Selection and Storage for Less Forgetting in Continual Learning](https://mlsanity.com/paper/2308.01895/) (2023)
- [Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions](https://mlsanity.com/paper/2401.04305/) (2024)
- [Bridging Streaming Continual Learning via In-Context Large Tabular Models](https://mlsanity.com/paper/2512.11668/) (2025)
- [How Relevant is Selective Memory Population in Lifelong Language Learning?](https://mlsanity.com/paper/2210.00940/) (2022)
- [Approximate Bayesian Class-Conditional Models under Continuous Representation Shift](https://mlsanity.com/paper/2305.19076/) (2023)
- [Continual Learning in Predictive Autoscaling](https://mlsanity.com/paper/2307.15941/) (2023)
- [Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocation](https://mlsanity.com/paper/2310.00154/) (2023)
- [How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning](https://mlsanity.com/paper/2407.07668/) (2024)
- [Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity](https://mlsanity.com/paper/2504.20932/) (2025)
- [Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?](https://mlsanity.com/paper/2309.10149/) (2023)
- [Bridging Neuroscience and AI: Environmental Enrichment as a Model for Forward Knowledge Transfer](https://mlsanity.com/paper/2405.07295/) (2024)
- [Task-agnostic Lifelong Robot Learning with Retrieval-based Weighted Local Adaptation](https://mlsanity.com/paper/2410.02995/) (2024)

## Abstract (arXiv)

A challenging problem in task-free continual learning is the online selection of a representative replay memory from data streams. In this work, we investigate the online memory selection problem from an information-theoretic perspective. To gather the most information, we propose the textitsurprise and the textitlearnability criteria to pick informative points and to avoid outliers. We present a Bayesian model to compute the criteria efficiently by exploiting rank-one matrix structures. We demonstrate that these criteria encourage selecting informative points in a greedy algorithm for online memory selection. Furthermore, by identifying the importance of textitthe timing to update the memory, we introduce a stochastic information-theoretic reservoir sampler (InfoRS), which conducts sampling among selective points with high information. Compared to reservoir sampling, InfoRS demonstrates improved robustness against data imbalance. Finally, empirical performances over continual learning benchmarks manifest its efficiency and efficacy.